I recently needed to install Visual Studio 2022 on one my existing machines to debug a new zeroshot model that has a dependency on our Speech SDK. The Speech SDK is one of our key #AI services in Cognitive Services (as part of #AzureAI). I already had VSCode running, but in this case I need the bigger brother.
After installing Visual Studio, I could not get any nuget packages to install; I could not even fetch anything and didn’t matter what I used – the package manager console in Visual Studio, PowerShell, etc.
Nuget would only point to the local offline package store, which in my case is available at: C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\. I kept getting the error: Package X is not found on source 'C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\'
I could not understand the behavior, and not seen this before. Turns out it is a known issue, albeit not very common. Seems like if either PowerShell’s Install-Module, or Chocolatey’s coco command was used before using nuget or Visual Studio for the first time, this will happen.
The solution is to add nuget.org as a package source, and the URL it should point to is: https://api.nuget.org/v3/index.json. Below is the screenshot on what this looks in Visual Studio for me:
Updated nuget package sources
If one is setting up a brand new dev box, the odds of you running into this is low; in my case given I was adding Visual Studio 2022 much later is what caused this. There is more details on this here too.
If you have a Tesla, then you should absolutely check out TeslaMate which is data logger for your car(s) that one self-hosts. This uses the car’s API and gets all different kinds of telemetry of your drives, charging, batter conditions, acceleration, braking, parking, etc. I personally prefer this, over other online services (of which there are a few) – as it is giving away the keys to the kingdom – literally in this case (the Tokens used to authenticate and login).
I have been running TeslaMate at home on a couple of machines for a while and figured a cloud version would work out better. I had network issues on one of the machines, where no car telemetry was downloaded. It was a few days before I realized that the machine wasn’t online due to a separate DNS issue and those few days’ worth of car telemetry (drives and other data of course) wasn’t recorded.
In our example, we will deploy TeslaMate in a docker container running on Ubuntu – which is hosted on Azure. To help with isolation and managing this, I would recommend we use a resource group (RG) only for running TeslaMate. Of course, we need an Azure subscription, which I would assume you already have.
We start by logging into the Azure portal and create a new resource group (RG) for TeslaMate; if you are not sure how to do this, the documentation here outlines the steps needed. Once you have an empty RG, it would look something like the screenshot below.
New Azure Resource Group
Step 2 – Creating new Ubuntu VM
Now that we have a new RG, we need to create a new Ubuntu virtual machine (VM) in that. We will choose the option to create resources as shown in the middle of the screen (see previous screenshot).
Clicking on “Create resources”, we see various menu options; the options you see might be a little different than the one shown below.
Create a resource – Azure
We need to create a Virtual Machine – the first choice under “Popular Azure services” and will click the “Create” link. This starts a wizard that allows you to go through the various settings and options.
The first step when creating a VM is to start with the basic details for machine we are creating – instance details, subscription details, admin user details, etc. I outline the steps and show screenshots to help those who are not comfortable with this level of tech, or new to Azure. If you are a more advanced user, a more efficient way would be via the Azure CLI. You can read up more details on VM’s on Azure here.
Step 3 – VM Basics
Subscription and resource group – Make sure you have the correct subscription and RG selected. If you haven’t created a new RG yet, you can do so using the “Create new” link under the RG option (see the screenshot below).
VM Name – You can give the VM any name – this is more for you to remember and manage.
Region – In terms of a region, in most cases it would make sense to pick a region that is physically close to the same area where you are based (and the car too of course).
Image – I use the latest Ubuntu LTS image which as of this writing is v20.04 Gen 2.
Create a new VM in Azure
Size – In terms of picking the size for the VM – we don’t need a very beefy machine, and needless to say – the bigger the machine, the higher the monthly costs. I keep the standard Size. This is not my main instance as I already have that running – this new instance is being setup as a demo that I will be deleting later.
Username – This is obvious and should be something you know and can remember.
Password – I choose password as the auth type, more so as this is for demo purposes for this post; ideally ssh keys are more secure and you would want to use that. If you do go down a password path, I cannot stress enough not to reuse passwords and create a strong password; it is always a good idea to use a password manager (e.g., I use LastPass).
Basic details required when setting up a new VM
I chose the simple SSD option; we don’t need a lot of advanced things.
Disk details when setting up a new VM
For the network options, you do want a public IP and, in most cases, just leaving the default would work. And I don’t show it in the screenshot, but we don’t need a load balancer and leave the default option of “None”. And we do want the ability to ssh into the machine to deploy and manage TeslaMate.
Networking details when setting up a new VM
For the Inbound port rules, by default only port 22 is enabled for SSH; to allow us to access the web server we also need to both ports 80 (http) and 443 (https) are enabled as shown in the screenshot below.
Inbound port rules – when setting up a new VM
For the next set of Tabs (Management, Advanced, and Tags) I didn’t change anything and went with the defaults. Once the validations are passed, and the final review shows the cost and other details you choose.
VM Creation – summary VM Creation – summary VM Creation – summary
And once you are happy with everything click the Create button on the bottom left corner.
VM Creation confirmation
Once the deployment of the VM starts, it can take a few minutes and you will see a similar progress as shown below.
VM deployment status screen
Once the VM is created, deployed and wired up (which can take a few minutes) – we will see the confirmation as shown below.
VM deployment confirmation
From the confirmation screen, clicking on “Go to resource” takes us to a screen where we see the different details of the VM. One of the details we are interested in at this point is the IP address and the ability to give the machine a DNS name. We need these to be able to connect to the VM over SSH (see screenshot below).
VM essential details
It might be worthwhile to also setup a DNS name that one can use in addition to the IP. This DNS name would be the fully qualified domain name (FQDN) that would be needed later when configuring the docker container. The DNS name allows us to connect to the machine using something like “https://my-car-details.cloudapp.azure.com (or similar). You can read more details on FQDN in the Azure docs here. If you are interested in using your own DNS server, you can read details on how to go about that here.
Click on the “Not Configured” for the DNS name (as shown in the image below) and you can set a unique name that is something memorable.
VM DNS name
The DNS name is tied to the region you have, and it must be unique.
VM DNS name label creating
And once this is setup, you can see the FQDN in your VM details as shown below.
VM DNS name
If for some reason you didn’t open ports 80 and 443 earlier, you can always configure them now. To do so, in the Azure portal, when you have the Ubuntu VM resource selected, click on Networking on the left, and you can update the Inbound port rules.
VM networking settingVM adding inbound network rules
When you add both ports (you would need to give them unique names and priority orders), and the final results would look something like the screenshot shown below.
VM network inbound port rules
And finally, we can ssh into that machine using the credentials and the IP we configured earlier. This can be done using ssh (e.g. ssh user-name@IP-address of the machine).
Step 4 – Install Docker
The first thing we need to do once we ssh into the machine is to update the various packages installed. The first time you run this, it will take a few minutes. You do this by running the following commands.
# I prefer to run these separately - to get a handle on what is getting updated.
sudo apt update
sudo apt upgrade
# You can of course run them together if that is what you prefer
sudo apt-get update
This is pretty standard and should not cause any issues; below is the screenshot showing the output – there are too many packages being updated for me to show everything.
Installing docker on our Ubuntu VM isn’t terribly complex – the docker docs outline all the steps and the details. We will want to install from the repro and follow the steps outlined and be mindful of specific versions and drivers.
We setup the repository, and for that need to install the following prerequisites.
At this time, we should run apt-get update command to update the newly added repository. We should check to ensure that docker is going to be installed from the docker repository, and not Ubuntu’s default. To do this we run the following command.
apt-cache policy docker-ce
This shows us that docker isn’t installed, but the candidate for installation is from docker.com and is for “focal” – which is the release name of Ubuntu v20.04. The list we see is long because it outlines the different versions of docker.
Now, we are finally ready to install docker using the following command and also choosing Yes on the prompts that confirm the installation.
Once complete, you will see something like the output shown below.
At this time, docker should be installed running. We can also check its daemon is configured to run on booting up.
Whilst not needed, it is good practice to add the current username to the docker group created – this will ensure we don’t need to use “sudo” for every docker command. And using the groups command we can validate our current username (“amit” in my case) is in the docker group.
sudo usermod -aG docker ${USER}
su - ${USER} # this allows us to add the user without logging out
Woot! We have docker running.
The first thing one should do with any new docker installation is to run its equivalent of Hello World. This is done using the following command – which downloads a test image and runs it in a container, prints a message, and then exists the container – so a full life cycle.
sudo docker run hello-world
And yay, we validated that docker is up and running on our VM! Congratulations.
Before we get to configuring TeslaMate, we also need to install docker-compose, which is a tool that allows us to run multi-container docker applications (such as TeslaMate). We will install docker-compose using the following command with the result of that command shown after that.
sudo apt install docker-compose
Step 5 – Configure TeslaMate
Given we will be exposing TeslaMate to the internet directly we should not use the default TeslaMate docker installation, but the advanced version which uses Traefik as a proxy server and helps us secure the web server better and only expose the (Grafana) dashboards behind an authentication mechanism.
For this we will create a new folder for TeslaMate which will contain not only the docker compose file needed but other relevant configuration details. I like to keep this in a folder, to help manage – in this case it resides in ~/docker/teslamate
It is in this folder we will create the docker-compose yaml file that is needed; you would want to start with the one outlined in the TeslaMate instructions and tweak it for your needs.
This file needs to be called docker-compose.yml and my example is shared below. It is a good idea to always get the latest yaml file from TeslaMate’s docs – over time we would expect things will evolve and the file below might not be accurate down the road.
Next we need to create a .env file. The environmental secrets (i.e. passwords) are not saved in the yaml file but are stored are stored in this .env file which we will create next.
We will enter the DNS name as the FQDN that you setup earlier for the VM; update the TM_TZ for the time-zone you are based out of. This is the TZ database name, and if you aren’t sure what it should be for your time-zone, check out the details here.
If you are not sure on how to create a new file in Ubuntu (or any other Linux distro for that matter) – you can use nano editor as shown below. You need to make sure this is in the same folder as where the docker-compose.yml file is (which is ~/docker/teslamate in our example).
Console screenshot showing how to create .env file
And finally, we need to create a .htpasswd file which is used to authenticate the website (see TeslaMate’s documentation for more details). I chose to create this locally after installing Apache tools, but you can also do it online. Note: this is *not* your Tesla login credentials but are the credentials you will use to access the site we are setting up now.
Console screenshot showing installation of Apache Utils
We can create a new file password as shown below. Given we are in the TeslaMate folder, we don’t have to provide a full path for the file.
htpasswd -c .htpasswd amit
Screenshot showing htpasswd usage
So, in the end we should have the following three files in the same folder:
Screenshot showing director listing
Step 6 – Starting TeslaMate
Now we are finally ready to start the docker container for TeslaMate. When this launches, go to the URL for the DNS name you setup, and login using your Tesla credentials. For the first time, I would recommend running the container attached to the console, so if there are any errors or issues you can see them. Normally you would want to run this detached (which is using the “-d“) option.
# don't forget the sudo command
sudo docker-compose up
The first time you run this, it will take a few minutes to pull all the images, and wire things up. During the process you will see the progress for each image in the various container app.
Console showing docker-compose progress
And finally, if everything is setup properly you should see the container running with the logs in the console of your terminal. This is a running log, and the process is active. You will see something like the screenshot below.
Console showing docker-compose logs
If you didn’t setup a DNS name earlier and thought you can try and use the IP name – that unfortunately will fail with the Traefik proxy server and in the logs, you will see an error to that effect. Let’s Encrypt doesn’t issue certificates for IP addresses as a policy.
Now if we browse the URL (also known as the FQDN) you had setup earlier, you will see an authentication challenge. This is great and shows that the proxy server is setup properly and working as expected.
Traefik http authentication
Once you enter the credentials you setup in the .htpasswd file earlier you will be able to login and see the TeslaMate’s Tesla login!
TeslaMate Tesla authentication screen
Congratulations! You have TeslaMate running on Azure.
Step 7 – Finishing up TeslaMate configuration
Now that you have TeslaMate running, you need to login to Tesla. The best way to do this these days is using existing tokens . There are a few ways to do this, and one of the easiest is using this tool – Tesla API Token.
Once you login, go to Settings and change the Dashboards URL – which would be your FQDN with “/grafana” appended. Remember the credentials you use for the dashboards (i.e. Grafana) are the ones you set in the .env file.
TeslaMate URL configuration
Now that everything is up and running, we can kill the docker-compose process, which is attached to the console, and re-run it detached from the console. To do this, go back to the ssh session we have connected to the Ubuntu VM and press CTRL + C. This will stop that container and you will see a similar output as shown below.
Console output
And now you can restart the container with the “-d” option, which is for detached.
sudo docker-compose up -d
docker-compose output
Congratulations, you have TeslaMate running on an Azure host Ubuntu VM via docker.
Below is a screenshot of my instance that has been running for some time.
It is 2021. And we have #AI writing #AI code. 🤪 It is quite interesting, but also can be quite boring once you get beyond the initial technology, and just think of it as one of the tools in your arsenal. And getting to that point is a good think.
As part of a think at work I recently started playing with GitHub Copilot, which is using GPT3 to be your pair programmer — helping write code. GPT3 has multiple models (called engines), and Copilot uses one of these family of engines called Codex. Codex is a derivative of the base GPT3 engine that is trained on billions of lines of code.
Using Copilot is quite simple; you install the Github Copilot extension, and it shows up in your IDE (VSCode in my example). We need to make sure we decompose the problem we are trying to solve – we should not think of this as helping write the complete program or all parts; but as it can help with different functions and pieces of code. To do this, we need to tell it what we are trying to do – these are done via prompts (code comments). For GPT models, prompt engineering is quite critical, and would be worth getting to details and understanding.
Starting simple, I create an empty python file and entered a prompt that outlines what I want to try and do. In this case as you can see in the screenshot below – I want to load an image from a file, and using our Vision Cognitive Services, run an image analysis, and auto-generate a caption for that image.
I started typing the definition of a function, and Copilot (via the add-in) understands the prompt I outlined, and the context of the code on what I am doing. Remember Codex builds on the base GPT3 and does have all that NLU capability.
Taking all of this in, it suggests completing the function for me. In terms of using this as an end-user (i.e. the developer) – the suggested code shows up as auto-complete and you can see it in the grey color. If I like that suggestion, I press tab and have it added to the file.
In this case you can see how it is reading the file from disk, calling a function called get_caption() and printing the caption to the stdout (console in this example).
There is also an option to cycle through different suggestions and then pick another one as shown in the screenshot below.
This variant of the suggested code is creating a function called image_caption() which takes the path to the image file to load. This also expects other required things for the Vision cognitive service to work – such as the subscription key to authenticate, the API end-point details to call, etc.
Typically, Copilot can synthesize up to 10 code options (Copilot calls these as Solutions), that one can cycle through and see if there is a better variant for the task at hand. The screenshot below shows this experience in VS Code.
The ask whilst simple, still involves a bit of code which needs to be written – reading from file, setting up the subscription details, and wiring that up to call this etc. And it is in these cases really where Copilot shines – it is your copilot picking up the ‘gunk work’ – freeing up your bandwidth, and cognition capacity on the more interesting and higher order bit of code and value to your business.
To get a flavor of our AI writing AI code, below is the full set of the nine suggestions the Copilot came back for what I was trying to do.
Synthesizing 9/10 solutions
=======
def generate_caption(image_path):
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from msrest.authentication import CognitiveServicesCredentials
from azure.cognitiveservices.vision.computervision.models import TextOperationStatusCodes
from azure.cognitiveservices.vision.computervision.models import TextRecognitionMode
from azure.cognitiveservices.vision.computervision.models import VisualFeatureTypes
from msrest.exceptions import CognitiveServicesError
from os.path import exists
from os import remove
from os import makedirs
from os import listdir
from os.path import isfile, join
from time import sleep
from sys import exit
from random import randint
from datetime import datetime
# set up the connection to the cognitive services account
subscription_key = "YOUR KEY HERE"
endpoint = "https://YOUR ENDPOINT HERE"
group_id = "YOUR GROUP ID HERE"
client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))
# set up the directory where the images will be downloaded
download_dir = "./downloads/"
if not exists(download_dir):
makedirs(download_dir)
# set up the directory where the images will be saved
save_dir = "./images/"
if not exists(save_dir):
makedirs(save_dir)
# set up the directory where the images will be saved
caption_dir = "./captions/"
if not exists(caption_dir):
makedirs(caption_dir)
# get a list of the files in the download directory
files = [f for f in listdir(download_dir) if isfile(join(download_dir, f))]
if len(files) == 0:
print("No files found in the download directory")
exit(1)
# get the first file in the list
file_name = files[0]
file
=======
def generate_caption(image_path):
from os import environ
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from msrest.authentication import CognitiveServicesCredentials
from azure.cognitiveservices.vision.computervision.models import TextOperationStatusCodes
from azure.cognitiveservices.vision.computervision.models import TextRecognitionMode
from azure.cognitiveservices.vision.computervision.models import VisualFeatureTypes
from msrest.exceptions import CognitiveServicesError
# Add your Computer Vision subscription key to your environment variables.
if 'COMPUTER_VISION_SUBSCRIPTION_KEY' in environ:
subscription_key = environ['COMPUTER_VISION_SUBSCRIPTION_KEY']
else:
print("\nSet the COMPUTER_VISION_SUBSCRIPTION_KEY environment variable.\n**Restart your shell or IDE for changes to take effect.**")
sys.exit()
# Add your Computer Vision endpoint to your environment variables.
if 'COMPUTER_VISION_ENDPOINT' in environ:
endpoint = environ['COMPUTER_VISION_ENDPOINT']
else:
print("\nSet the COMPUTER_VISION_ENDPOINT environment variable.\n**Restart your shell or IDE for changes to take effect.**")
sys.exit()
# Create an authenticated Computer Vision client.
client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))
# Open the local image file.
with open(image_path, 'rb') as image_file:
# Read the image into a byte array.
image_data = image_file.read()
# Set the features to return.
features = [VisualFeatureTypes.categories, VisualFeatureTypes.description, VisualFeatureTypes.tags]
# Analyze the local image.
try:
result = client.analyze_image(image=image_data, visual_features=features)
=======
def generate_caption(image_file, subscription_key):
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from msrest.authentication import CognitiveServicesCredentials
from PIL import Image
from io import BytesIO
client = ComputerVisionClient(ComputerVisionClient.SUBSCRIPTION_KEY_ENDPOINT,
CognitiveServicesCredentials(subscription_key))
image_stream = open(image_file, 'rb')
image = Image.open(image_stream)
image_bytes = BytesIO(image)
analysis = client.analyze_image(image_bytes, ComputerVisionClient.DESCRIPTION)
return analysis.description.captions[0].text
=======
def generate_caption(image_filename):
import os
import json
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from msrest.authentication import CognitiveServicesCredentials
from PIL import Image
from io import BytesIO
import sys
import requests
# Set the environment variables for your subscription key and region
# For example, for US West
# "COMPUTER_VISION_SUBSCRIPTION_KEY=<subscription key>"
# "COMPUTER_VISION_ENDPOINT=https://westus2.api.cognitive.microsoft.com/vision/v2.0"
# For example, for China East
# "COMPUTER_VISION_SUBSCRIPTION_KEY=<subscription key>"
# "COMPUTER_VISION_ENDPOINT=https://eastasia.api.cognitive.microsoft.com/vision/v2.0"
# Set your subscription key and region
subscription_key = os.environ['COMPUTER_VISION_SUBSCRIPTION_KEY']
endpoint = os.environ['COMPUTER_VISION_ENDPOINT']
# Set your image path
image_path = image_filename
# Set your region
region = "eastus"
# Create an instance of the Computer Vision client
computer_vision_client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))
# Load the image from disk
with open(image_path, 'rb') as image_file:
image_content = image_file.read()
# Add the image to the list of images to analyze
image_list = [image_content]
# Make the analysis asynchronous
analyze_result = computer_vision_client.analyze_image_in_stream(image_list, visual_features=['Categories', 'Description', 'Color'])
# Get the analysis result
analysis = analyze_result.result()
# Display the image captions
print("\nImage captions:")
for caption in analysis.description.
=======
def generate_caption(image_path):
# load the image
image = open(image_path, "rb").read()
# encode the image into base64
image = base64.b64encode(image).decode("utf-8")
# create a dict with the image
body = {"image": image}
# create a dict with the subscription key
headers = {"Ocp-Apim-Subscription-Key": "0d5a5f7f1e534d05a8e8d6f9e6e9c6d3"}
# create a url for the api
url = "https://westcentralus.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=Description,Tags"
# send the request
response = requests.post(url, headers=headers, json=body)
# get the response
response.raise_for_status()
# get the caption
caption = response.json()["description"]["captions"][0]["text"]
# return the caption
return caption
=======
def generate_caption(image_path):
from os import environ
from os.path import join
from urllib.parse import urlparse
from urllib.parse import parse_qs
from http.client import HTTPSConnection
from json import loads
from base64 import b64encode
# Load the image into a byte array
image_data = open(image_path, "rb").read()
image_data = b64encode(image_data).decode('utf-8')
# Read the subscription key from the environment variables
subscription_key = environ["AZURE_SUBSCRIPTION_KEY"]
# Define the vision API endpoint
vision_base_url = "https://westus2.api.cognitive.microsoft.com/vision/v2.0/"
# Set the vision API endpoint to analyze the image
vision_analyze_url = vision_base_url + "analyze"
# Set the parameters
parameters = {'visualFeatures': 'Categories,Description,Color'}
# Prepare the request
headers = {'Ocp-Apim-Subscription-Key': subscription_key,
'Content-Type': 'application/octet-stream'}
params = parse_qs(urlparse(vision_analyze_url).query)
params['visualFeatures'] = 'Categories,Description,Color'
params['language'] = 'en'
# Send the request
conn = HTTPSConnection('westus2.api.cognitive.microsoft.com')
conn.request("POST", vision_analyze_url, image_data, headers)
response = conn.getresponse()
data = response.read()
# Load the JSON response into a Python dictionary
parsed = loads(data)
return parsed
=======
def generate_caption(image_path):
"""
Generate a caption for an image.
:param image_path: path to an image
:return: the caption
"""
# Load the image
from os.path import isfile
from PIL import Image
import requests
if not isfile(image_path):
raise FileNotFoundError(image_path)
image = Image.open(image_path)
# Encode the image
image_data = open(image_path, "rb").read()
# Computer Vision parameters
params = {'visualFeatures': 'Categories,Description,Color'}
# Computer Vision API URL
url = "https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze"
# Connect to Computer Vision
headers = {'Ocp-Apim-Subscription-Key': 'd3f3e1b8e7e14d0b9d7d3f8c8f7e0b6f', 'Content-Type': 'application/octet-stream'}
response = requests.post(url, headers=headers, params=params, data=image_data)
response.raise_for_status()
# Load the response as JSON
analysis = response.json()
# Extract the word tags
categories = [(category['name'], category['score'])
for category in analysis['categories']]
print('Tags:')
for category in categories:
print(category[0])
# Extract the description
descriptions = [(description['text'], description['score'])
for description in analysis['description']['captions']]
print('Descriptions:')
for description in descriptions:
print(description[0])
# Display the color
try:
dominant_color = analysis['color']['dominantColorBackground']
except:
dominant_color = 'N/A'
print('Dominant color: {}'.format(dominant_color))
# Display the caption
caption = '{}'.format(
=======
def generate_caption(filename):
import os
import sys
import json
import requests
import cognitive_services_keys
import cognitive_services_key_names
# create the header for the request
headers = {'Ocp-Apim-Subscription-Key': cognitive_services_keys.get_key(cognitive_services_key_names.BING_KEY)}
# the URL for the bing endpoint
bing_url = 'https://api.cognitive.microsoft.com/bing/v5.0/images/search'
# the parameters for the request
params = {'q': 'image', 'count': 1, 'offset': 0, 'mkt': 'en-us', 'safeSearch': 'Moderate'}
# read the contents of the file
with open(filename, 'rb') as f:
data = f.read()
# build the request
request = requests.post(bing_url, headers=headers, params=params, data=data)
# check for errors
if request.status_code != 200:
print('Error code: {}'.format(request.status_code))
print(request.text)
sys.exit(1)
# process the response
response = json.loads(request.text)
if 'value' in response:
caption = response['value'][0]['name']
print('The caption is: {}'.format(caption))
else:
print('No caption found.')
=======
def generate_caption(image_path):
import os
import json
import requests
subscription_key = "e9e9c8a8d1d9400e8d0b7f2d8f8e7c6e"
assert subscription_key
vision_base_url = "https://westcentralus.api.cognitive.microsoft.com/vision/v1.0/"
analyze_url = vision_base_url + "analyze"
image_data = open(image_path, "rb").read()
headers = {'Ocp-Apim-Subscription-Key': subscription_key,
'Content-Type': 'application/octet-stream'}
params = {'visualFeatures': 'Categories,Description,Color'}
response = requests.post(
analyze_url, headers=headers, params=params, data=image_data)
response.raise_for_status()
analysis = response.json()
print(analysis)
image_caption = analysis["description"]["captions"][0]["text"].capitalize()
return image_caption
I been kicking the tires with Open AI’s #GPT-3. Based on the screenshot below, it might be easy to think “oh boy does the model think highly of itself”, but as with most things in life – devil is in the details.😃 The screenshot below was a forked version of davinci engine and follows the Q&A structure.
GPT-3 vs other AI assistants
Using OpenAI’s API is quite simple; perhaps too simple! It is quite easy to unleash the beast as the code snippet shown below. If you are new to using GPT3, I would highly recommend you start with the use case model guidelines first.
In the context of a toy example, to get to a simple Q&A chatbot as the screenshot earlier shown is quite simple. The API is powerful, and simple to use, and getting started is easy as the code below shows.
import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
response = openai.Completion.create(
engine="davinci",
prompt="I am a highly intelligent question answering bot. If you ask me a question that is rooted in truth, I will give you the answer. If you ask me a question that is nonsense, trickery, or has no clear answer, I will respond with \"Unknown\".\n\nQ: What is human life expectancy in the United States?\nA: Human life expectancy in the United States is 78 years.\n\nQ: Who was president of the United States in 1955?\nA: Dwight D. Eisenhower was president of the United States in 1955.\n\nQ: Which party did he belong to?\nA: He belonged to the Republican Party.\n\nQ: What is the square root of banana?\nA: Unknown\n\",
temperature=0,
max_tokens=100,
top_p=1,
frequency_penalty=0.0,
presence_penalty=0.0,
stop=["\n"]
)
There are three core concepts when using GPT-3: Prompt, Completion, and Tokens.
To start using the API, we need to start giving it some prompts – this provide some context to the engine on what is expecting. Without the surface area is too broad and we get into nonsensical situations. This is part of the task-specific fine-tuning required.
Think of when giving examples as part of the prompt, we are essentially “programming” the model and providing guidance and providing some hints to context and pattern matching. Note the training data cut off in late 2019, so the model in production today doesn’t have access to data and events post that (e.g., Covid).
Completion is the output that GPT3 generates based on the prompt. To be clear, this is not the full text but is the predicted completions; think of it as “autocomplete” in Word, or Outlook or a search engine. The API has flexibility to return more than one predicted completion along with the probabilities of alternative tokens at each position (to me it seems just like the wave function when thinking of Quantum mechanics 🐼).
Finally, think of Token are the smaller Lego blocks that combine to make words. The API, which is nothing but wrappers around GPT-3 breaks up the text into tokens before processing it. The GPT-3 model understands the statistical relationships between these tokens and uses this to produce the next token in a sequence of tokens.
For example, if we are curious about Tokens, we can see in the screenshot below how the API “tokenizes” this paragraph and get the details of the tokens. This paragraph contains 207 characters and 43 tokens.
GPT-3 Tokens – TextGPT-3 Token – IDs
At a high level, think of one token == ~4 characters of text, which is ¾ of a word; so, 100 tokens ~= 75 words.
This is just dipping our toes in the beast that is GPT-3; the API’s which wrap up and expose the engines (more on that in another post) make it simple to use and without getting too much in the weeds of 175 billion parameters. 🙂
If you are like me and get annoyed with the big PowerShell upgrade ‘nag’ ‘reminder’ (see screenshot below); instead of trying to figure out what to download and install the update, there is a simpler way to get the latest update and address the nag. 🙂
You can just run the code below in an elevated prompt to get the latest release of PowerShell – it is easy-peasy. 🙂
If you are like me, and don’t really have your work saved in the “%USERPROFILE%” it gets annoying after a time, to keep changing the directory.
If there is one specific folder that you prefer, it is an easy configuration change in the profile setting – add a setting called “startingDirectory” and point it to the path you want.
For example, I have a root folder called “src” where most of the code I am working on sits, and that’s where I wanted to default the terminal to.
To get to the profile, you can either use the shortcut CTRL+, or from the dropdown in the title bar, click settings (see below). This will open the settings.json in your default editor.
Terminal setting
In my case, I wanted the starting directory for all the shells, so I put it under “defaults” – you can choose different options for different shells, and then would have this in the appropriate shell’s settings and not the default block of course.
Below is what this looks like for me pointing this to “c:\src”. Also note, the escape characters need to be formatted properly to parse.
"defaults":
{
// Put settings here that you want to apply to all profiles.
"fontFace": "CaskaydiaCove NF",
"startingDirectory": "c:\\src",
},
setting.json screenshot
Once you save the file, it should automatically reload the terminal. And if the json didn’t parse – because of a typo or a syntax error then you would see an error similar to the one shown below.
Parsing error
In this example, I set the starting folder as “c:\src”; instead of “c:\\src”.
I recently needed to get a list of users that belong to a specific Microsoft Teams team – and there isnt anything out of the box to get this using the Teams app. AFAIK, the only way to do this is using the Microsoft graph API – for which there are a few options.
For something quick (e.g. getting a list of users in a team), using the Graph explorer could be easy enough. On the other hand, if you need something more robust, you should program against the (REST) API.
Graph Explorer
Navigate to Graph explorer, sign in and authenticate yourself against the specific O365 tenant you are interested in – most folks would only have one.
Microsoft Graph Explorer
Once authenticated, on the panel on the left you see several sample queries and scroll down until you see the Teams.
Teams sample queries
To get members of a specific team, you need to get the team ID for that Team. This is unique ID (GUID) and doesn’t change over the lifetime of the team. If you have this, then go ahead to the next section – Getting team members.
Getting a list of Teams and Team ID
On the query panel in Graph explorer, select the “my joined teams” and run the query. You will get a JSON back that contains the list of teams that you are a member of. The “id” element represents the Team ID which you would need for any team related API calls. For example, I am interested in this specific #AI team: “#Reinforcement Learning and Decision AI”.
Get team details
Getting team members
Once you have the Team ID (the unique GUID that each identifies each team), you can get the members of the team using that option on the left. As shown on the screenshot below, you do need to pass in the team ID to the REST API and this would be something like this (and don’t worry what I am showing below is a fictious GUID):
If you want something more robust and repeatable, then using the API (via code) or PowerShell might be better. If you are programming, you will need to register an app – which can authenticate using the Identify platform. This of course is quite powerful, but at times for simple things might be a bit too much.
In my simple task to get users from Teams, I prefer the PowerShell option. To get this going first you need to install the MicrosoftTeam module. This can be done using the command below.
Install-Module -Name MicrosoftTeams
Depending on your configuration you might get a warning as shown below.
PowerShell module installation
Once the Teams PowerShell module is installed, you can run PowerShell scripts against Teams and achieve the same result. I have two scripts below showing the same steps as with the Graph Explorer above. One of these gets details of the teams that a user is a member of. And the second script is to get members of a selected team.
Using PowerShell to get Team Details
The PowerShell script below to get a Team details is below; you can also get it from GitHub. Before you run this, there are two variables that need to be set.
One, the path where you want the team details to be exported (this is a csv file).
Two, set the email that you will use. This needs to be the same one that you authenticated against.
You will be prompted to sign in to authentic and this should be an experience that most folks would be familiar with. Note, each time you run the script, you need to authenticate – and this is irrespective of say if you are already logged into Teams of Office 365.
Authenticating user against Office 365
Assuming you have authenticated successfully, you should see an output like the one shown below; and a csv file in the path you configured will be created. This file will always be overwritten – without any prompts (of course this is assuming no other process is open that has a lock on that file).
#Set these variables, to what makes sense in your situation. The email here is the one that is the one connected to your teams account.
$exportLocation = "C:\temp\team-details.csv"
$emailAddress = "your-email@shouldbeputhere.com"
#Authenticate against teams
Connect-MicrosoftTeams
#Patience
Write-Host -ForegroundColor Blue "Successfully connected to Teams"
Write-Host -ForegroundColor Blue "Getting all team details for user: $($emailAddress)"
Write-Host -ForegroundColor Blue "Please be patient, if there are a lot of teams, this can take a while..."
# Get all of the team Groups IDs
# $GetUsersTeams = (Get-Team).GroupID
$GetUsersTeams = Get-Team -User $emailAddress
$Report = @()
# Will hold a basic count of user types and teams
$unavailableTeamCount = 0
# Loop through all teams that the user belongs to
$currentIndex = 1
ForEach($thisTeam in $GetUsersTeams) {
# Show some output to the user
Write-Progress -Id 0 -Activity "Building report from Microsoft Teams" -Status "$currentIndex of $($GetUsersTeams.Count)" -PercentComplete (($currentIndex / $GetUsersTeams.Count) * 100)
# Attempt to get team details, throw error message if no access
try {
# Get team members
#$users = Get-TeamUser -GroupId $thisTeam.groupID
# Create an object to hold all values
$teamReportObject = New-Object PSObject -Property @{
GroupID = $thisTeam.GroupID
TeamName = $thisTeam.DisplayName
Description = $thisTeam.Description
Archived = $thisTeam.Archived
Visibility = $thisTeam.Visibility
eMail = $thisTeam.MailNickName
}
# Add to the report
$Report += $teamReportObject
} catch [Microsoft.TeamsCmdlets.PowerShell.Custom.ErrorHandling.ApiException] {
Write-Host -ForegroundColor Yellow "No access to $($team.DisplayName) team, cannot generate report"
$unavailableTeamCount++
}
$currentIndex++
}
Write-Progress -Id 0 -Activity " " -Status " " -Completed
# Disconnect from teams
Disconnect-MicrosoftTeams
# Provide some nice output
Write-Host -ForegroundColor Green "============================================================"
Write-Host -ForegroundColor Green " Microsoft Teams User Report "
Write-Host -ForegroundColor Green ""
Write-Host -ForegroundColor Green " Count of All Teams - $($GetUsersTeams.Count) "
Write-Host -ForegroundColor Green " Count of Inaccesible Teams - $($unavailableTeamCount) "
Write-Host -ForegroundColor Green ""
$Report | Export-CSV $exportLocation -NoTypeInformation -Force
Write-Host -ForegroundColor Blue "Exported report to $($exportLocation)"
Getting Team members using PowerShell
Now that you have the Team ID you are interested, you can run the other PowerShell script (also available on GitHub) to get a list of all the users in a specific team. Like the previous script, you would need set a couple of variables in the script:
The Team ID for the team you are interested in.
Path for the csv file with details to be saved.
Once you have authenticated and ran the script, the output will look like the one shown below. You get a summary of the team details, and details of the Teams users and their type (owner, member, or guest). And just like earlier, the file will be overwritten without a prompt, assuming no locks on it.
Members of a Microsoft Team
#Global variables to set:
#path of the file where to export
#specific ID of the team that you want the users for.
$exportLocation = "C:\temp\RL-decision-AI-export.csv"
$TEAM_ID = "f3f9ad1f-beea-4026-9b86-dd3788404999"
$Report = @()
# counters
$ownerCount = 0
$memberCount = 0
$guestCount = 0
#connect to teams
Connect-MicrosoftTeams
$team = Get-Team -GroupId $TEAM_ID
#Patience, supposed to be a virtue
Write-Host -ForegroundColor Blue "Successfully connected to Team: $($team.DisplayName)"
Write-Host -ForegroundColor Blue "Getting all users in the team"
Write-Host -ForegroundColor Blue "Please be patient, if there are a lot of users, this can take a while..."
# Attempt to get team users, throw error message if no access
try {
# Get team members
$users = Get-TeamUser -GroupId $team.groupID
# Loop through and get all the users
$currentIndex = 1
# foreach user create a line in the report
ForEach($user in $users) {
# Show some output to the user
Write-Progress -Id 0 -Activity "Generating user report from Teams" -Status "$currentIndex of $($users.Count)" -PercentComplete (($currentIndex / $users.Count) * 100)
# Maintain a count of user types
switch($user.Role) {
"owner" { $ownerCount++ }
"member" { $memberCount++ }
"guest" { $guestCount++ }
}
# Create an object to hold all values
$ReportObject = New-Object PSObject -Property @{
User = $user.Name
Email = $user.User
Role = $user.Role
}
# Add to the report
$Report += $ReportObject
$currentIndex++
}
}
catch [Microsoft.TeamsCmdlets.PowerShell.Custom.ErrorHandling.ApiException] {
Write-Host -ForegroundColor Yellow "No access to $($team.DisplayName) team, cannot generate report"
}
#Complete progress
Write-Progress -Id 0 -Activity " " -Status " " -Completed
# Disconnect from teams
Disconnect-MicrosoftTeams
# Write out details for the user
Write-Host -ForegroundColor Green "============================================================"
Write-Host -ForegroundColor Green " Microsoft Teams User Report "
Write-Host -ForegroundColor Green ""
Write-Host -ForegroundColor Green "Team Details:"
Write-Host -ForegroundColor Green "Name: $($team.DisplayName)"
Write-Host -ForegroundColor Green "Description: $($team.Description)"
Write-Host -ForegroundColor Green "Mail Nickname: $($team.MailNickName)"
Write-Host -ForegroundColor Green "Archived: $($team.Archived)"
Write-Host -ForegroundColor Green "Visiblity: $($team.Visibility)"
Write-Host -ForegroundColor Green ""
Write-Host -ForegroundColor Green "Team User Details:"
Write-Host -ForegroundColor Green "Owners - $($ownerCount)"
Write-Host -ForegroundColor Green "Members - $($memberCount)"
Write-Host -ForegroundColor Green "Guests - $($guestCount)"
Write-Host -ForegroundColor Green "============================================================"
$Report | Export-CSV $exportLocation -NoTypeInformation -Force
Write-Host -ForegroundColor Blue "Exported report to $($exportLocation)"
Of course, programming against the API is always more powerful, but sometimes quick and easy is what is needed. 🙂
Been playing with a few things at home, and as part of that was trying to get Docker and Docker Compose running on a Raspberry Pi. Docker Compose if you aren’t familiar with, allows one to run multi-container apps, and is very handy when building multi-tier layered applications – which are quite common.
I was running it docker on my (Synology) NAS, but a recent update from them broke docker – specifically environment variables. That in turn broke the ability to run Docker Compose, and of course a bunch of stuff; and the opportunity to experiment.
First, we need to install docker – which these days is quite simple. You need the ability to ssh into the pi (or if you are connected to a display, then via a terminal prompt). And in some cases if things fail then you might need to run them as root (via sudo). To install docker, run the following:
curl -sSL https://get.docker.com | sh
And once you are done installing docker, then test it by running the classic hello world image. To so that you run the following command – this will get the Hello World image, and once run will automatically remove it (which is because of the –rm option)
docker run --rm hello-world
If everything is installed OK, then you should see a output that looks something like this the shown below. And this is good – means everything is up and running as expected.
pi@pi-server2:~ $ docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1eda109e4da: Pull complete
Digest: sha256:b8ba256769a0ac28dd126d584e0a2011cd2877f3f76e093a7ae560f2a5301c00
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(arm32v7)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
To make like more simple, you should add the user you are logged in as to the ‘docker’ group. In my case it is the default ‘pi’ user, so that command would look like this. And for this to take effect, you would need to logout – I just reboot the machine – old habits. 🙂
sudo usermod -aG docker pi
OK, now that docker is installed, lets get to docker-compose. For that we first install pip, and use that to install docker-compose. And don’t forget the apt-get update in the end.
Now before anything else, lets try and make sure all dependencies are there. Create a file called ‘docker-compose.yml’ with the following. You can put this file anywhere, but I like to create a separate folder and save it in that.
In this example I expose port 6666 to the host which is mapped to port 8000 internally on the image. If your port 6666 is taken you can choose another port – it doesn’t matter. Spacing and indent, do matter in a yml file, so you would want to pay extra attention to that.
Once the file is saved you run it with the following command. The image handles you would see are very likely going to be different and that is OK.
pi@pi-server2:~/docker/docker-test $ docker-compose up
Creating network "docker-test_default" with the default driver
Pulling webapp (python:3.7-alpine)...
3.7-alpine: Pulling from library/python
33b18ff7f9b7: Pull complete
0c1f90421c3a: Pull complete
91543a0ba590: Pull complete
913b1310b79e: Pull complete
6b545e90ee55: Pull complete Digest: sha256:9363cb46e52894a22ba87ebec0845d30f4c27efd6b907705ba9a27192b45e797
Status: Downloaded newer image for python:3.7-alpine
Creating docker-test_webapp_1 ... done Attaching to docker-test_webapp_1
At this point, the image is running in attached mode and it seems like it is waiting, when in reality it is running. If you open another ssh terminal and type in the following command – change the port to whatever you used earlier in the yml file.
pi@pi-server2:~ $ curl -iv 0.0.0.0:6666
And if everything is working then you will see a output something like this. And if you see towards the top you got a HTTP 200 – that is all that mattes in this case.
* Expire in 0 ms for 6 (transfer 0x1b097c0)
* Trying 0.0.0.0...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x1b097c0)
* Connected to 0.0.0.0 (127.0.0.1) port 6666 (#0)
> GET / HTTP/1.1
> Host: 0.0.0.0:6666
> User-Agent: curl/7.64.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
HTTP/1.0 200 OK
< Server: SimpleHTTP/0.6 Python/3.7.4
Server: SimpleHTTP/0.6 Python/3.7.4
< Date: Thu, 26 Sep 2019 22:10:28 GMT
Date: Thu, 26 Sep 2019 22:10:28 GMT
< Content-type: text/html; charset=utf-8
Content-type: text/html; charset=utf-8
< Content-Length: 915
Content-Length: 915
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".dockerenv">.dockerenv</a></li>
<li><a href="bin/">bin/</a></li>
<li><a href="dev/">dev/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="home/">home/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="media/">media/</a></li>
<li><a href="mnt/">mnt/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="proc/">proc/</a></li>
<li><a href="root/">root/</a></li>
<li><a href="run/">run/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="srv/">srv/</a></li>
<li><a href="sys/">sys/</a></li>
<li><a href="tmp/">tmp/</a></li>
<li><a href="usr/">usr/</a></li>
<li><a href="var/">var/</a></li>
</ul>
<hr>
</body>
</html>
* Closing connection 0
You can go back to the first ssh session and hit Ctrl + C to shutdown the image. Once you do that you will see something like:
^CGracefully stopping... (press Ctrl+C again to force)
Stopping docker-test_webapp_1 ... done pi@pi-server2:~/docker/docker-test $
Now you know docker-compose and all the dependencies are installed. Next I would want docker to auto start whenever the pi boots up, and for that we will use the following two commands.
Haven’t had time until now to explore on what is new as Tesla continues to push updates. The latest version as of this post is v3.9.1 which is what there I decompiled and when compared to the earlier version (I had posted (v3.8.2), there three new REST API’s outlined below.
Service data from the car – not sure what exactly does this will. Need to try it.
Now, when I call that, I get a 200OK response (see below), so it is accepting the request, and that includes the bearer code in the header as expected. I don’t see anything interesting back, but that could be because my car is not in service. Maybe someone who has their vehicle in the service center can try and validate this.
{
"response": {}
}
The next new API is a POST, for reports; and calling this just sends a 200OK back, but I don’t know what it is for. It seems very similar to the SEND_LOG method.
The next two set of APIs seem quite interesting and related t AutoPilot upgrade. It might be that these could be in app purchases – checking the eligibility, and then allowing one to purchase.
When I try and call the Purchase_URL, I get a HTTP 400, and seems like I am missing some parameters – other than the headers.
{
"error": "bad_request",
"error_description": "The data given to this server does not meet our criteria."
}
And calling the eligible_upgrades I get a ‘false’. Now I already have AutoPilot, so this might make sense. And given this seems to be a key-value pair, I am guessing there will be other things that Tesla would add over time to up-sell.
{
"autopilot": false
}
The final new API is related to energy sites, and something I of course don’t have or have an interest, but sharing here if someone does care. 🙂
This is a output of a few frustrating hours (spanning over a few days – as and when I can get time), and finally got it fixed and working. Hopefully it might help someone who is also dealing with npm blues.
When NodeJS and npm works, its awesome. But when it borks, it is worst than my code or so it seems :).
Been playing with a few things and wanting to get a dashboard going with Grafana (and InfluxBD as a time-series DB). But some of the installation was failing and for the life of me, could not figure out why and how. Clean image install and downgrading to the previous stable version also didn’t help.
One example of npm failing miserably was the “Error: Method not Allowed” which is not very helpful. Here is an example of what I was seeing:
root@pi-server:/var/lib/grafana/plugins/grafana-trackmap-panel# npm install
(node:4538) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
npm ERR! Error: Method Not Allowed
npm ERR! at errorResponse (/usr/share/npm/lib/cache/add-named.js:260:10)
npm ERR! at /usr/share/npm/lib/cache/add-named.js:203:12
npm ERR! at saved (/usr/share/npm/node_modules/npm-registry-client/lib/get.js:167:7)
npm ERR! at FSReqWrap.oncomplete (fs.js:135:15)
npm ERR! If you need help, you may report this *entire* log,
npm ERR! including the npm and node versions, at:
npm ERR! <http://github.com/npm/npm/issues>
npm ERR! System Linux 4.19.57-v7+
npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install"
npm ERR! cwd /var/lib/grafana/plugins/grafana-trackmap-panel
npm ERR! node -v v8.11.1
npm ERR! npm -v 1.4.21
npm ERR! code E405
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /var/lib/grafana/plugins/grafana-trackmap-panel/npm-debug.log
npm ERR! not ok code 0
root@pi-server:/var/lib/grafana/plugins/grafana-trackmap-panel#
Again, like I said not very helpful. But I finally got to be able to fix it and move on. And here is what worked for me, and it seems like in the OS image, there was a corrupted files, at some level. In most cases you need root access.
A key virtue of a programmer is laziness. As an example it is what inspires me to automate my home to the point where I don’t have to lift a finger to switch on the light. Removing friction from a system is a anesthetic joy. The drug of efficiency, feels really good.
I still write code and people get surprised by that sometimes – maybe it’s the quality of the code 🤓.
I have been playing with a #selfdriving car for a while, and that is super exciting. From a #AI and #ML perspective it is small scale, but allows one to exploit all aspects of the tech stack and also appreciate the limitations of not only the software, but also the hardware.
With this You run a NN on a raspberry pi that uses TensorFlow, and Keras and runs inference on the edge. The pi doesn’t have enough power to train, so you need to do that on a beefier machine and then deploy the model back to run this.
Now, I didn’t have any issues in getting this running on Windows, but to get it on a Mac was a different story. The documentation is there that outlines all the steps, and even if you follow it to the T, it breaks right in the end.
When I tried to create a car, using a createcar command (this essentially creates the buckets, where you would save the training images, and the model, and the configuration of the car when you connect to it from your machine). The actual file paths would probably be different for you but, essentially it is the same thing.
(donkey) AMAC02XN1T9JGH5:donkeycar amit.bahree$ donkey createcar ~/mycar
Traceback (most recent call last):
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 660, in _build_master
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 968, in require
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 859, in resolve
pkg_resources.ContextualVersionConflict: (imageio 2.4.1 (/anaconda3/envs/donkey/lib/python3.6/site-packages), Requirement.parse('imageio<3.0,>=2.5'), {'moviepy'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/anaconda3/envs/donkey/bin/donkey", line 6, in <module>
from pkg_resources import load_entry_point
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2985, in <module>
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2971, in _call_aside
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2998, in _initialize_master_working_set
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 662, in _build_master
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 675, in _build_from_requirements
File "/anaconda3/envs/donkey/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 854, in resolve
pkg_resources.DistributionNotFound: The 'imageio<3.0,>=2.5' distribution was not found and is required by moviepy
The key here to focus is on the last lines on both of those blocks of code – the main thing causing the issue is MoviePy (see highlighted lines above).
MoviePy is a Python library for video editing: cutting, concatenations, title insertions, video compositing (a.k.a. non-linear editing), video processing, and creation of custom effects.
It seems like when you go through the steps – clone the repo, setup anaconda, install tensorflow and get the car configured – there is a mismatch in the MoviePy dependencies which it doesn’t like. The way to fix the issue is outlined below.
Skip MoviePy
MoviePy is something you don’t need to use right away but later when trying to make a movie (using the makemovie command – which allows you to create a movie file from the images in a Tub.); this is not essential. To do this, the easiest way is to remove (or my suggestion it to comment) out the moviepy dependency from the setup.py file.
This should be line 33 in the setup.py file that you will find in the same folder where you cloned the git repo. As an example the updated file is below, where the moviepy dependency is commented out (see highlighted). And once you save this and go about creating the car, it should work. Of course you cannot use the makemovie option later.
from setuptools import setup, find_packages
import os
with open("README.md", "r") as fh:
long_description = fh.read()
setup(name='donkeycar',
version='2.5.7',
description='Self driving library for python.',
long_description=long_description,
long_description_content_type="text/markdown",
url='https://github.com/autorope/donkeycar',
download_url='https://github.com/autorope/donkeycar/archive/2.1.5.tar.gz',
author='Will Roscoe',
author_email='wroscoe@gmail.com',
license='MIT',
entry_points={
'console_scripts': [
'donkey=donkeycar.management.base:execute_from_command_line',
],
},
install_requires=['numpy',
'pillow',
'docopt',
'tornado==4.5.3',
'requests',
'h5py',
'python-socketio',
'flask',
'eventlet',
#'moviepy',
'pandas',
],
extras_require={
'tf': ['tensorflow>=1.9.0'],
'tf_gpu': ['tensorflow-gpu>=1.9.0'],
'pi': [
'picamera',
'Adafruit_PCA9685',
],
'dev': [
'pytest',
'pytest-cov',
'responses'
],
'ci': ['codecov']
},
include_package_data=True,
classifiers=[
# How mature is this project? Common values are
# 3 - Alpha
# 4 - Beta
# 5 - Production/Stable
'Development Status :: 3 - Alpha',
# Indicate who your project is intended for
'Intended Audience :: Developers',
'Topic :: Scientific/Engineering :: Artificial Intelligence',
# Pick your license as you wish (should match "license" above)
'License :: OSI Approved :: MIT License',
# Specify the Python versions you support here. In particular, ensure
# that you indicate whether you support Python 2, Python 3 or both.
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
],
keywords='selfdriving cars donkeycar diyrobocars',
packages=find_packages(exclude=(['tests', 'docs', 'site', 'env'])),
)
Once you have saved the setup.py file, you need to run the installation again with the following command and then run the create car command. Both of these are outlined below.
pip install -e .
donkey createcar ~/mycar
Once you run these, then you should see the successful installation as shown by the output below. Note – your output might be a little different depending on the conda state of packages
(donkey) AMAC02XN1T9JGH5:donkeycar amit.bahree$ pip install -e .
Obtaining file:///Users/amit.bahree/CloudStation/Documents/Code/donkeycar
Requirement already satisfied: numpy in /anaconda3/envs/donkey/lib/python3.6/site-packages (from donkeycar==2.5.7) (1.14.5)
Requirement already satisfied: pillow in /anaconda3/envs/donkey/lib/python3.6/site-packages (from donkeycar==2.5.7) (4.2.1)
Requirement already satisfied: docopt in /anaconda3/envs/donkey/lib/python3.6/site-packages (from donkeycar==2.5.7) (0.6.2)
Collecting tornado==4.5.3 (from donkeycar==2.5.7)
Requirement already satisfied: requests in /anaconda3/envs/donkey/lib/python3.6/site-packages (from donkeycar==2.5.7) (2.18.4)
Requirement already satisfied: h5py in /anaconda3/envs/donkey/lib/python3.6/site-packages (from donkeycar==2.5.7) (2.7.1)
Collecting python-socketio (from donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/a1/71/118e4b7fb453d7095d6863f4b783dbaa57109af4bc2380300649c8942d61/python_socketio-4.0.0-py2.py3-none-any.whl
Collecting flask (from donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl
Collecting eventlet (from donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/86/7e/96e1412f96eeb2f2eca9342dcc4d5bc9305880a448b603b0a8e54439b71c/eventlet-0.24.1-py2.py3-none-any.whl
Collecting pandas (from donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/99/12/bf4c58eea94cea4f91ff931f284146337814fb8546e6eb0b52584446fd52/pandas-0.24.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Requirement already satisfied: olefile in /anaconda3/envs/donkey/lib/python3.6/site-packages (from pillow->donkeycar==2.5.7) (0.44)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /anaconda3/envs/donkey/lib/python3.6/site-packages (from requests->donkeycar==2.5.7) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /anaconda3/envs/donkey/lib/python3.6/site-packages (from requests->donkeycar==2.5.7) (2017.7.27.1)
Requirement already satisfied: idna<2.7,>=2.5 in /anaconda3/envs/donkey/lib/python3.6/site-packages (from requests->donkeycar==2.5.7) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /anaconda3/envs/donkey/lib/python3.6/site-packages (from requests->donkeycar==2.5.7) (1.22)
Requirement already satisfied: six in /anaconda3/envs/donkey/lib/python3.6/site-packages (from h5py->donkeycar==2.5.7) (1.10.0)
Collecting python-engineio>=3.2.0 (from python-socketio->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/95/91/d083bd7b5d408af53633377dfbf87bf181236c8916d36213388b12eaa999/python_engineio-3.4.3-py2.py3-none-any.whl
Collecting click>=5.1 (from flask->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Collecting itsdangerous>=0.24 (from flask->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting Werkzeug>=0.14 (from flask->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl
Collecting Jinja2>=2.10 (from flask->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting monotonic>=1.4 (from eventlet->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/ac/aa/063eca6a416f397bd99552c534c6d11d57f58f2e94c14780f3bbf818c4cf/monotonic-1.5-py2.py3-none-any.whl
Collecting greenlet>=0.3 (from eventlet->donkeycar==2.5.7)
Collecting dnspython>=1.15.0 (from eventlet->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/ec/d3/3aa0e7213ef72b8585747aa0e271a9523e713813b9a20177ebe1e939deb0/dnspython-1.16.0-py2.py3-none-any.whl
Collecting pytz>=2011k (from pandas->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl
Collecting python-dateutil>=2.5.0 (from pandas->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask->donkeycar==2.5.7)
Using cached https://files.pythonhosted.org/packages/f0/00/a6aea33f5598b080b86d6b6d1214b51afe3ffa6100b902d5aa465080083f/MarkupSafe-1.1.1-cp36-cp36m-macosx_10_6_intel.whl
Installing collected packages: tornado, python-engineio, python-socketio, click, itsdangerous, Werkzeug, MarkupSafe, Jinja2, flask, monotonic, greenlet, dnspython, eventlet, pytz, python-dateutil, pandas, donkeycar
Found existing installation: tornado 4.5.1
Uninstalling tornado-4.5.1:
Successfully uninstalled tornado-4.5.1
Found existing installation: Werkzeug 0.12.2
Uninstalling Werkzeug-0.12.2:
Successfully uninstalled Werkzeug-0.12.2
Running setup.py develop for donkeycar
Successfully installed Jinja2-2.10 MarkupSafe-1.1.1 Werkzeug-0.14.1 click-7.0 dnspython-1.16.0 donkeycar eventlet-0.24.1 flask-1.0.2 greenlet-0.4.15 itsdangerous-1.1.0 monotonic-1.5 pandas-0.24.1 python-dateutil-2.8.0 python-engineio-3.4.3 python-socketio-4.0.0 pytz-2018.9 tornado-4.5.3
And when I run the createcar, you can see it worked as expected. In my case creating the ‘mycar’ folder in my home directory. Of course you can choose this wherever you prefer.
(donkey) AMAC02XN1T9JGH5:donkeycar amit.bahree$ donkey createcar ~/mycar
using donkey version: 2.5.7 ...
Creating car folder: /Users/amit.bahree/mycar
making dir /Users/amit.bahree/mycar
Creating data & model folders.
making dir /Users/amit.bahree/mycar/models
making dir /Users/amit.bahree/mycar/data
making dir /Users/amit.bahree/mycar/logs
Copying car application template: donkey2
Copying car config defaults. Adjust these before starting your car.
Donkey setup complete.
It is interesting to see this is more stable on Windows, than on a Mac. Also, one last thing to leave you with – when I first ran the installation, the hint that someone was wrong was in the output, but I didn’t pay too much attention to it. See the red line highlighted in the output below.
moviepy failure – donkeycar installation
Don’t know at this time on what the solution for moviepy is to get this sorted – luckily its not a big deal at the moment.
As my experimentation continues, I wanted to get Visual Studio Code installed on a mac, and wanted to use python as the language of choice – main reason for the mac is to understand and explore the #ML libraries, runtimes, and their support on a mac (both natively and in containers – docker).
Now, Microsoft has a very nice tutorial to get VSCode setup and running on a mac, including some basic configuration (e.g. touchbar support). But when it comes to getting python setup, and running, that is a different matter. Whilst the tutorial is good, it doesn’t actually work and errors out.
Below is the code that Microsoft outlines in the tutorial for python. It essentially is the HelloWorld using packages and is quite simple; but this will fail and won’t work.
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(0, 20, 100) # Create a list of evenly-spaced numbers over the range
plt.plot(x, np.sin(x)) # Plot the sine of each x point
plt.show() # Display the plot
When you run this, you will see an error that is something like the one outlined below.
The main reason this fails is that one has to be a little more explicit with matplot (the library that we are trying to use). Matplot has this concept of backends, which essentially is the runtime dependencies needed to support various execution environments – including both interactive and non-interactive environments.
For matplot to work on a mac, the raster graphics c++ library that it uses is based on something called Anti-Grain Geometry (AGG). And for the library to render, we need to be explicit on which agg to use (there are multiple raster libraries).
In addition on a mac OS X there is a limitation when rendering in OSX windows (presently lacks blocking show() behavior when matplotlib is in non-interactive mode).
To get around this, we explicitly tell matplot to use the specific agg (“TkAgg in our case) and then it will all work. I have a updated code sample below, which adds more points, and also waits for the console input, so one can see what the output looks like.
import matplotlib
matplotlib.use("TkAgg")
from matplotlib import pyplot as plt
import numpy as np
def waitforuser():
input("Press enter to continue ...")
return
x = np.linspace(0, 50, 200) # Create a list of evenly-spaced numbers over the range
y = np.sin(x)
print(x)
waitforuser()
print(y)
waitforuser()
plt.plot(x,y)
plt.show()
And incase you are wondering what it looks like, below are a few screenshots showing the output.
Sometime back, I noticed the car downloaded a large file (5.1 GB) which was a .ssq file. I hadn’t heard of a ssq file, and was curious on what this was.
I researched a little and as it turns out, a .ssq file is a compressed file system which is often used in an embedded Linux system, where storage size might be a area of concern. This file-system is called SquashFS, and is usually used on a read-only mode.
SquashFS is interesting, as it lets one mount the file-system directly and is distributed as a kernel source patch – which makes it easy to daisy chain and use it other regular Linux tools.
SquashFS tools are useful to mount and create a SquashFS file-system. As shown below, I can mount the downloaded file, using unsquashfs.
When extracted and mounted, we see the following directory structure; each of these folders (and files therein) are in fact the tiles that make up the maps (next time in the car, when you zoom in or out or search of a non-cached location, notice carefully on how it is loading and you can just about make out the tiles – it is quick and easy to miss). And it is these tiles that is used for routing as part of the navigation.
Tiled based routing is supposed to be beneficial – it uses less memory (the graph can be decomposed much easier, with a smaller set of it loaded in memory), cahce-able, easier to manage (update-able), etc. We can see a glimpse on how the routing and calculation happen on a tile basis below.
tiles based routing
When, extracted we see there are three levels of hierarchy (0, 1, and, 2). In the file-system these are shown as directories, but there is a method to the madness.
Level 0 – these contain edges pertaining to roads that are considered highway / freeway / motorway roads. These are stored as 4 degree tiles.
Level 1 – contains roads that are at a arterial level and are saved in 1 degree tiles.
Level 2 – these are local roads and are saved as 0.25 degree tiles.
For example, the world at Level 0 would look like what we are seeing in the image below. And Pennsylvania can be seen below that; Level 0 colored in light blue, Level 1 in light green, and finally Level 2 in light red (which might not be obvious with the translucency).
World Level 0 tilesPennsylvania Level 0, 1, and 2 tiles
So, to use this, one can use a few helper functions to get the exact tile to load and vice-versa. For example using the GPS coordinate of 41.413203, -73.623787 (which is just outside of Brewster, NY), loading Level 2 (via the get_title_2 function) would give us the structure of /2/000/756/425.gph using which we know which tile to load.
Helper function (in python) that help obtain levels, tile ids, tile lists, lat/long coordinates, etc. from an intersecting box.
valhalla_tiles = [{'level': 2, 'size': 0.25}, {'level': 1, 'size': 1.0}, {'level': 0, 'size': 4.0}]
LEVEL_BITS = 3
TILE_INDEX_BITS = 22
ID_INDEX_BITS = 21
LEVEL_MASK = (2**LEVEL_BITS) - 1
TILE_INDEX_MASK = (2**TILE_INDEX_BITS) - 1
ID_INDEX_MASK = (2**ID_INDEX_BITS) - 1
INVALID_ID = (ID_INDEX_MASK << (TILE_INDEX_BITS + LEVEL_BITS)) | (TILE_INDEX_MASK << LEVEL_BITS) | LEVEL_MASK
def get_tile_level(id):
return id & LEVEL_MASK
def get_tile_index(id):
return (id >> LEVEL_BITS) & TILE_INDEX_MASK
def get_index(id):
return (id >> (LEVEL_BITS + TILE_INDEX_BITS)) & ID_INDEX_MASK
def tiles_for_bounding_box(left, bottom, right, top):
#if this is crossing the anti meridian split it up and combine
if left > right:
east = tiles_for_bounding_box(left, bottom, 180.0, top)
west = tiles_for_bounding_box(-180.0, bottom, right, top)
return east + west
#move these so we can compute percentages
left += 180
right += 180
bottom += 90
top += 90
tiles = []
#for each size of tile
for tile_set in valhalla_tiles:
#for each column
for x in range(int(left/tile_set['size']), int(right/tile_set['size']) + 1):
#for each row
for y in range(int(bottom/tile_set['size']), int(top/tile_set['size']) + 1):
#give back the level and the tile index
tiles.append((tile_set['level'], int(y * (360.0/tile_set['size']) + x)))
return tiles
def get_tile_id(tile_level, lat, lon):
level = filter(lambda x: x['level'] == tile_level, valhalla_tiles)[0]
width = int(360 / level['size'])
return int((lat + 90) / level['size']) * width + int((lon + 180 ) / level['size'])
def get_ll(id):
tile_level = get_tile_level(id)
tile_index = get_tile_index(id)
level = filter(lambda x: x['level'] == tile_level, valhalla_tiles)[0]
width = int(360 / level['size'])
height = int(180 / level['size'])
return int(tile_index / width) * level['size'] - 90, (tile_index % width) * level['size'] - 180
Tesla has actually open-sourced their implementation of Valhalla, which is based on C++. This still seems like an active project, but parts of the code haven’t been updated for a while.
Whilst I haven’t tried to set this up myself, it seems quite simple. Below are the instructions to get this going on Ubuntu or Debian (I think Mac is also supported, but needs a little different dependency set).
#below are the dependencies needed
sudo add-apt-repository -y ppa:valhalla-core/valhalla
sudo apt-get update
sudo apt-get install -y autoconf automake make libtool pkg-config g++ gcc jq lcov protobuf-compiler vim-common libboost-all-dev libboost-all-dev libcurl4-openssl-dev libprime-server0.6.3-dev libprotobuf-dev prime-server0.6.3-bin
#if you plan to compile with data building support, see below for more info
sudo apt-get install -y libgeos-dev libgeos++-dev liblua5.2-dev libspatialite-dev libsqlite3-dev lua5.2
if [[ $(grep -cF xenial /etc/lsb-release) > 0 ]]; then sudo apt-get install -y libsqlite3-mod-spatialite; fi
#if you plan to compile with python bindings, see below for more info
sudo apt-get install -y python-all-dev
#install with the following
git submodule update --init --recursive
./autogen.sh
./configure
make test -j$(nproc)
sudo make install
There you have it – we know now what the .ssq files are and how they are used. Just need more time to get it going and play with it – perhaps another project for another time. 🙂
In case you haven’t been following the news, Tesla is in the process of releasing the new firmware beta. I think many folks online are super interested in new autopilot upgrades.
I reverse engineered the associated app and there are certainly a few new end points exposed, as outlined below. Need time to now figure out more details on this and what they entail. Also need time to see what changes in the existing code and json (data structure).
Is it interesting to go noodle on this, and see the associated calls. This outlines all the products as of today
Some time ago, I talked about my Tesla Model 3 “keyfob” which essentially uses a Amazon IoT button to call some of Tesla API’s and “talk” to the car. This for me, is cool as it allows my daughter to unlock, and lock the car at home. And of course it is a bit geeky, and allowing one to play with more things. 🙂
Since publishing this, I was surprised how many of you ping me asking on details on how they can did this for themselves. Given the level of interest, I thought I will document this and outline the steps here. I do have to warn you, that this would be a little long – it entails getting a IoT Button configured, and then the code deployed. Before you get started, and if you aren’t techy, I would recommend to go through the post completely, so you get a sense of what is needed.
At a high level, below are the steps that you need to go through to get this working. And this might seem cumbersome and a lot but it is not that difficult. Also if you prefer you can follow the official AWS documentation online here.
Create a AWS Login (if you have a existing Amazon.com login, you can use the same one if you prefer)
Order a IoT Button
Register the IoT Button in the AWS Registry (this is done via the AWS console)
Create (and activate) a device certificate
Create a IoT security policy
Attach the IoT security policy (from the previous step) to the device certificate created earlier
Attach the IoT security policy (now with the associated certificate) to the IoT button
Configure the IoT button
Deploy some code – this is done via a server-less function (also called a Lambda function) – this is the code that gets executed
Open AWS home page and login with your amazon.com credentials. Of course if you don’t have a Amazon.com account, then you want to click in sign up on the top right corner, to get this started.
AWS Login
After I login, I see something similar to the screenshot below. Your exact view might differ a little.
AWS Console
I recommend to change the region to one closer to you. To do this, click on the region on the top right corner and choose a region that is physically closest to you. In the longer run this would help with latency issues between you clicking the button and the car responding. For example in my case, Oregon makes most sense.
AWS Region Selection
Once you have a AWS account setup, login to the AWS IoT console or on the AWS page in the previous step, scroll down to IoT Core as shown in the screenshot below.
AWS Console
Step 3 – Register IoT Button
Next step would be to register your IoT button – which of course means you physically have the button with you. The best way to register is to follow the instructions here. I don’t see much sense in trying to replicate that here.
Note: If you are not very technical, or comfortable, it might be best to use either the “AWS IoT Button Dev” app which is available both on the Apple Store (for iOS) and Google play (for Android).
Once you have registered a button (it doesn’t matter what you call it) – it will show up similar to the screenshot below. I only have one device listed.
List of IoT things
Step 4 – Create a Device Certificate
Next, we need to create and activate a certificate for the device. Without this, the button won’t work. The certificate (which is a X.509 certificate) protects the communication between the button and AWS.
For most people, the one-click certification creation that AWS has, is probably the way to go. To get to this, on the AWS IoT console, click on Secure and then choose Certificates on the left if not already selected as shown below. I already have a certificate that you can see in the screenshot below.
Certificates
If you need to create a certificate, click on the Create button on the top right corner, and choose one of the options shown in the image below. In most cases you would want to use the One-click certificate option.
Certificate creation options
NOTE: Once you create a Certificate, you get three files (these are the keys) that you need to download and keep safe. The certificate itself can be downloaded anytime, but the private and the public keys CANNOT be retrieved again after you close this page. It is IMPORTANT that you download these and save them in a safe place.
Certificate Keys
Once you have these downloaded then click on Activate on the bottom. And you should see a different certificate number than what you are seeing here. And don’t worry I have long deleted what you are seeing on this screen. 🙂
You can also see these in the developer guide on AWS documentation.
Step 5 – Create a IoT Security Policy
Next step is go back to the AWS IoT Console page and click on Policies under Security. This is used to create a IoT policy that you will need to attach to the certificate. Once you have a policy created, then it will look something like the screenshot below.
IoT Policies
To create a policy, click on Create (or you might be prompted automatically if you don’t have one). On the create screen, in the Name you can enter anything that you prefer. I would suggest naming this something that you can remember and differentiate if you will have more than one button. In my case I named it as the same thing as my device.
In the policy statements for Action enter “iot:Connect” – without the quotes, but this is case sensitive so make sure you match is exactly.
For the Resource ARN enter “*” (again without the quotes) as shown below.
And finally for the effect, make sure “Allow” is checked.
And click on Create at the bottom.
IoT Policy Creation
After this is created this you will see the policies listed as shown below. You can see the new one we just created with “WhateverNameYouWillRecognize“. You can also see these and more details on the developer documentation – Create a AWS IoT Policy.
IoT Policies
Step 6 – Attach a IoT Policy
Next step is to attach the policy that is just created to the certificate created earlier. To do that, click on Secure and Certificates on the left, and then click on the three dots (called ellipses) on the top right of the Certificate you created earlier. From the new menu that you get, choose “Attach Policy” as shown below.
Attach Policy to Certificate
From the resulting menu, select the policy that you had created earlier and select Attach. Using a sensible name that you would recognize would be helpful. You can also see these details on the developer documentation.
Attach Policy to Certificate
Step 7 – Attach Certificate to IoT Device
Next step is to attach the certificate to the IoT device (or thing). A device must have a certificate, a private key and a root CA certificate to authenticate with AWS. Amazon also recommends to attach a device certificate to the device – this probably isn’t helpful right now, but might be in the future if you start playing with this more.
To do this, select the certificate under Security on the left, and same as the previous step, by click on the three dots on the top right corner, select “Attach thing”.
Attach Certificate
And from the next screen select the IoT button that you registered earlier, and select “Attach”.
Attach Certificate
Step 8 – Configure IoT Button
To validate that everything is setup correctly – the certificate needs to be associated with a policy, and a thing (the IoT button in our case). So on the Certificates menu on the left, select your certificate by clicking on it (not the three dots this time – but rather the name). You will see a new screen that shows the details of the certificate as shown below.
Certificate Details
And on the new menu on the
left, if you click on Policies you should see the policy you created, and
the Things should have the IoT button you created earlier.
Once all of this is done the next step is to configure the device. You can see more detailed steps on this on the developer guide here.
KEY TIP: The documentation doesn’t make it too obvious, but as part of configuring – the device (IoT Button) will become an access point that you will need to connect to and upload the certificates and key you created earlier. You cannot do this from a phone and it is best done from a desktop/laptop that has wifi network. Whilst these days all laptops will have a wifi network card, that isn’t necessarily true for desktops. So use a machine which has a wifi that you can temporarily connect to the access point that the IoT device creates.
Note this is only needed for getting the device configured to authenticate for AWS, and get on your Wifi network; once that is done you don’t need to do this.
At last we are starting to get the interesting part – a lot of what we were doing until now, was getting the button configured and ready.
Now that you have a IoT button configured and registered, the next step is to deploy some code. For this you need to setup a Lambda function using the AWS Lambda Console.
When you login, click on Create Function. On the Create function screen, choose the Blueprints option as shown below. You can see some of these in the developer documentation here.
Create Function screen
Step 10 – Blueprint Search
On the Blueprints search box (which says Filters by tags), type in “button” (without quotes) and press enter. You should see an option called “iot-button-email” as shown below, select that and click configure on the bottom right corner.
IoT Button filter
Step 11 – Basic Information
On the next screen that says “Basic information”, enter the details as shown below. The names should be meaningful for you to remember. Roles can be reused across other areas, for now you can use a simple name something like “unlockCar” or “unlockCarSomeName” if you have more than one vehicle. The policy template should already be populated and you shouldn’t need to do anything else.
Function basic information
For the 2nd half – AWS IoT Trigger, select the IoT type as “IoT Button” and enter your device serial number as outlined in the screenshot below.
IoT Trigger
It won’t hurt to download these certificate and keys in addition to the ones created separately and save them in different folders. And for the Lambda function code, it doesn’t matter on the template code as we will be deleting it all. At this point that will be read-only and you won’t be able to modify anything – as shown in the screen shot below.
Lambda function
And finally scrolling down more, you will see the environment variables. Here is where you need to specify your Tesla credentials to it to be able to use create the token and call the Tesla API. For that you need the following two variables: TESLA_EMAIL and TESLA_PASS. These case sensitive so you need to enter them as is. And then finally click on Create function.
Environment Variables
Step 12 – Code upload
Once you create a
function, you will see something like the screen below. In my case the function
is called “unlockSquirty” which is what you are seeing. This is
divided in to two parts – when on the Configuration page. The top part is the
designer that visually shows you what inputs are the triggers that execute the
function, and then what it outputs to on the right hand side. And below the designer is the editor where
one can edit the code inline or upload a zip file with the code.
In the function code
section, on the first drop down in the left (Code entry type) select upload a
.zip file.
And on the next screen upload the function package that you can download from here.
Make sure the Runtime is Node.js 8.10
Keep the Handler as the default.
Double check your Environment variable contain TESLA_EMAIL, and TESLA_PASS.
And scroll down and in the Basic settings, change the timeout to 1 minute. We run thus asynchronously and adding a little buffer would be better. You can leave all the other settings at their default. If your network might be iffy you can make this 2 mins.
Environment Settings
Step 13 – Code Publish
Once you have entered all of this, click on Save on the top right corner and then publish new version. Finally once it is published you will be able to see the code show up as shown in the screenshot below.
Again, a single
click will unlock the car, a double-click would lock it, and a long press
(holding it for 2-3 seconds) would open the charge port door.
And here is the code:
var tjs = require('teslajs');
var username = process.env.TESLA_EMAIL;
var password = process.env.TESLA_PASS;
exports.handler = (event, context, callback) =>
{
tjs.loginAsync(username, password).done(function(result)
{
var token = JSON.stringify(result.authToken);
if (token)
console.log("Login Succesful!");
var options =
{
authToken: result.authToken
};
tjs.vehicleAsync(options).done(function(vehicle)
{
console.log("Vehicle " + vehicle.vin + " is: " + vehicle.state);
var options =
{
authToken: result.authToken,
vehicleID: vehicle.id_s
};
if(event.clickType == "SINGLE")
{
console.log("Single click, attempting to UNLOCK");
tjs.doorUnlockAsync(options).done(function(unlockResult)
{
console.log("Doors are now UNLOCKED");
});
}
else if(event.clickType == "DOUBLE")
{
console.log("Double click, attempting to LOCK");
tjs.doorLockAsync(options).done(function(lockResults) {
console.log("Doors are now LOCKED");
});
}
else if(event.clickType == "LONG")
{
console.log("Long click, attempting to CHARGE PORT");
tjs.openChargePortAsync(options).done(function(openResult) {
console.log("Charge port is now OPEN");
});
}
});
});
};
![bah ree
powershell 7.e.2
Copyright Cc) Microsoft Corporation .
https :// aka. ms/powershell
Type 'help' to get help.
All rights reserved.
A new PowerShell stable release is available: v7.€.3
Upgrade now, or check out the release page at:
https :// aka . 9.3
Loading personal and system profiles took 1€89ms.
+ ambahree@AMBAHREE-LAPTOP
) iex
$Cirm https :// aka.ms/install—powershell.psl) } —UseMSI"
VERBOSE: About to download package from 'https
e msi '
[€9 : 41]](https://i0.wp.com/oldblog.desigeek.com/wp-content/uploads/2020/10/image.png?resize=660%2C212&ssl=1)
