Download Folders from a GitHub Repo using Python (… files, too)

At some point during your life as a programmer, you will end up with the problem of downloading a single folder from a GitHub repository. For me, it was because I had an auxiliary repo containing files relevant to some unit tests. To solve this problem, I found a very elegant (and super easy) solution, which I want to share here today. In other words, this is a short how-to on how to download/copy files and folders from GitHub using python.

The solution uses the awesome fsspec library. fsspec is a pythonic approach to filesystem management, and allows you to use python to access data in various kinds of locations: on your local machine, on all major cloud providers, and – most importantly – on GitHub. There are many more locations, so the library is worth checking out if you have the time.

Installation

To get started, install fsspec. (Chances are you already have it, because increasing parts of the pydata ecosystem use it internally.)

pip install fsspec

Copy a Folders

import fsspec
from pathlib import Path
# flat copy
destination = Path.home() / "test_folder_copy"
destination.mkdir(exist_ok=True, parents=True)
fs = fsspec.filesystem("github", org="githubtraining", repo="hellogitworld")
fs.get(fs.ls("src/"), destination.as_posix())
# recursive copy
destination = Path.home() / "test_recursive_folder_copy"
destination.mkdir(exist_ok=True, parents=True)
fs = fsspec.filesystem("github", org="githubtraining", repo="hellogitworld")
fs.get(fs.ls("src/"), destination.as_posix(), recursive=True)
view raw folder.py hosted with ❤ by GitHub
Example of how to download a folder from GitHub (shallow or recursive).

The above snippet does the following: We first declare a destination (where to store the folder’s content). Then we use fsspec to turn the repo into a pythonic filesystem. Finally, we list all the files in the target folder of the repo (fs.ls(…)) and download them all using fs.get. Simple, elegant, and convenient. I love it!

Copy Files

Copying/Downloading individual files works the same way; however, this time the destination has to be a file.

import fsspec
from pathlib import Path
# download a single file
destination = Path.home() / "downloaded_readme.txt"
fs = fsspec.filesystem("github", org="githubtraining", repo="hellogitworld")
fs.get("README.txt", destination.as_posix())
view raw copy_file.py hosted with ❤ by GitHub
Example of how to download a single file from GitHub.

And that is all there is to it. Happy coding!

Advertisement

Webcam Capture in Python (without OpenCV)

Did you know that there is a python library that allows you to to capture both a webcam stream or a single webcam image? Did you know that this works on every OS? This is what I want to share in this post: A tutorial on how to use imageio to access your webcam on Linux, Windows, or MacOS that works in either a Python script or a Jupyter Notebook. No OpenCV needed 🙂

Before we begin, a caveat for Jupyter: While the notebook is displayed on your current machine, and widgets run locally, your kernel (that runs the python code) may be hosted on a remote server, docker container, virtual machine, … depending on your setup. If this is the case for you, please note that IPython can only access the remote server’s webcam, not your local one.

Installation

pip install imageio[ffmpeg]

Get a single Image/Screenshot

import imageio as iio
import matplotlib.pyplot as plt
camera = iio.get_reader("<video0>")
screenshot = camera.get_data(0)
camera.close()
plt.imshow(screenshot)
view raw webcam_screenshot.py hosted with ❤ by GitHub

Get a Video as a Sequence of Frames

import imageio as iio
import matplotlib.pyplot as plt
import time
camera = iio.get_reader("<video0>")
meta = camera.get_meta_data()
delay = 1/meta["fps"]
for frame_counter in range(15):
frame = camera.get_next_data()
time.sleep(delay)
camera.close()
plt.imshow(frame)
view raw webcam_stream.py hosted with ❤ by GitHub

Full example (record a short MP4)

import imageio as iio
import matplotlib.pyplot as plt
import time
camera = iio.get_reader("<video0>")
meta = camera.get_meta_data()
num_frames = 5 * int(meta["fps"])
delay = 1/meta["fps"]
buffer = list()
for frame_counter in range(num_frames):
frame = camera.get_next_data()
buffer.append(frame)
time.sleep(delay)
camera.close()
iio.mimwrite("frames.mp4", buffer, macro_block_size=8, fps=meta["fps"])
view raw record_webcam_video.py hosted with ❤ by GitHub

That’s it.

If you happen to run into problems, you have three options to ask for help:

  1. Ask a question on StackOverflow and tag it with imageio (I monitor the tag)
  2. Comment on this post
  3. Create a New Issue / Bug Report on GitHub.

Thanks for reading and Happy Coding!

SSO for your App via Auth0 + Nginx + Docker + Vouch-Proxy

This post is a tutorial on “How to setup SSO via Auth0 using Nginx and Vouch-Proxy”. I couldn’t find an existing nifty blog post on this; so I ended up having to figure it out. Here, I want to document the steps so that others (also future me) may have an easier time setting this up.

If you don’t want a full tutorial, and just look for an example, here is a link to the repository with the final config files: https://github.com/FirefoxMetzger/sso_example

The setup I am presenting here works on localhost, and is mainly aimed at local development. It is a Docker-based setup, so there are tons of existing tutorials for deployment. Another thing that you may want to look into is hardening (making things super secure). I left this part out (for the most part) to avoid distraction; I really just want to focus on getting SSO up and running.

Setup Auth0

The first step is to set up Auth0 and create a new tenant. Make sure to pick a region that is close to your physical location; this will affect the login speed, but also how the data you send to Auth0 will be handled (data laws).

Setup window for a new tenant (Oct 2020)

Currently (2020), this will create a default app and enable authentication via email/password and google as a social login provider. We will use this default app. You can of course customize, but I recommend you first set it up following this tutorial, and then add your customization afterward.

Next, we will navigate to the settings of the default app.

Navigate to the settings page.

There are a few useful items in the settings which we will need, but the first thing is to allow users of our app to log in and log out. For this, we need to tell Auth0 which URLs are okay to use as callbacks for both login (Allowed Callback URLs) and logout (Allowed Logout URLs).

Navigate to Application URIs. For Allowed Callback URLs add http://localhost/sso/auth, https://localhost/sso/auth, http://localhost:9090/auth . For allowed logout URLs add http://localhost, https://localhost, http://localhost/sso/validate, http://localhost:9090/validate .

Configuration of Login and Logout inside Auth0.

Make sure to hit save changes at the bottom of the page.

We will delete most of these URLs as we move along, and they mainly exist for testing (so that we can assemble this incrementally). The two HTTPS URLs are the final ones, that we will use when we are done. The URLs on port 9090 are for testing vouch-proxy, which by default runs on port 9090, and the remaining HTTP URLs are for testing nginx as a reverse proxy for vouch-proxy and your app.

While we are now done with the setup for Auth0, don’t leave the settings page yet. At the top of the page you can find the applications domain, client ID and the client secret. We will need this info in the next steps, so keep it around.

Client ID and Client Secret location.

Setup Vouch-Proxy

Vouch-Proxy can almost run out of the box and all we need to do is add a config file. It follows the example for a generic OIDC provider, which you can find on the vouch-proxy repo. I made some modifications to make it work with Auth0.

When you use this template, be sure to replace the Auth0 domain with your domain, replace the client ID with your client ID, and replace the client secret with your client secret.

# vouch config
# bare minimum to get vouch running with OpenID Connect (such as okta)
vouch:
logLevel: debug
testing: true
listen: 0.0.0.0
port: 9090
allowAllUsers: true
jwt:
secret: Your-64-character-secret-key-here
issuer: Vouch
compress: false
cookie:
name: my-vouch-ct
secure: false
domain: localhost
headers:
jwt: X-Vouch-Token
querystring: access_token
redirect: X-Vouch-Requested-URI
accesstoken: X-Vouch-IdP-AccessToken
idtoken: X-Vouch-IdP-IdToken
post_logout_redirect_uris:
– https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}&returnTo=http://localhost/
oauth:
# Generic OpenID Connect
# including okta
provider: oidc
client_id: {client_id_from_auth0}
client_secret: {client_secret_from_auth0}
auth_url: https://your-project-name-here.eu.auth0.com/authorize
token_url: https://your-project-name-here.eu.auth0.com/oauth/token
user_info_url: https://your-project-name-here.eu.auth0.com/userinfo
scopes:
– openid
– email
– profile
callback_url: http://localhost:9090/auth
view raw config.yml hosted with ❤ by GitHub
Example config.yml for vouch proxy

To store this file, in your project’s folder create a sub-folder named vouch and in it another one named config. The relative path (from the project root) to the file is ./vouch/config/config.yml. You can check the GitHub repo for reference.

Next, it is time to test if vouch-proxy can correctly communicate with Auth0. I promised a dockerized setup, so let’s create a docker-compose file. (We will expand this file later.)

version: '3'
services:
vouch:
image: voucher/vouch-proxy
volumes:
– ./vouch/config/config.yml:/config/config.yml
ports:
– 9090:9090
view raw docker-compose.yml hosted with ❤ by GitHub
Initial Docker-Compose config file.

Save the file in the project’s root directory as docker-compose.yml. Then, navigate to the root directory and bring up the “stack” with a

docker-compose up

Once it is up and running, you can open your browser and navigate to localhost:9090. You should be greeted with a 

404 page not found

This is expected, so don’t worry; it’s merely a test if the server is alive (page not found >> Site can’t be reached).

Next, we will test the login flow between vouch-proxy and Auth0. For this navigate to

http://localhost:9090/login?url=http://localhost:9090/validate

The url= parameter specifies the location that we want the user to return to after the login has completed. In this case, we navigate to /validate, which is the endpoint we will use throughout the app to validate the client’s access token.

Once you put that into your browser, you will be greeted by a simple HTML page telling you that you are being redirected to some address. This is vouch-proxy’s debug mode which lets you check if your flow works correctly. Click the long that forwards to Auth0.

Vouch Proxy testing page.

This should present you with Auth0’s login form. Here we want to create a new user with username and password and authenticate ourselves.

Signup Page at Auth0.

Once you have an account and have accepted the permissions, you will be redirected to vouch-proxy and it will confirm that you are logged in with your chosen email.

Vouch proxy indicated that the user is authorized.

Now the only thing to test is to logout the user. Here there are multiple options. (1) You can log out the user from your app (vouch-proxy), (2) log the user out of your app and Auth0, or (3) you can log the user out of your app, Auth0, and their social login provider (if they use a social login). We will not cover the third one here.

To use the first option navigate to

localhost:9090/logout

which will tell you that you have been logged out. At this point, you are still logged in at Auth0, so after logging back into the app ( http://localhost:9090/login?url=http://localhost:9090/validate ), you will not be asked to log in at Auth0.

To log yourself out of Auth0 in parallel with your app, you have to tell vouch-proxy to redirect the user to the logout URL of Auth0. You can read more about it here. To logout in both places, use the URL below.

http://localhost:9090/logout?url=https://dev-simple.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}%26returnTo=http://localhost:9090/validate

Be sure to replace {client_id_from_auth0} with your client ID. Also, notice the percent encoding of the ampersand (%26), which, if left out, will break the logout procedure. If you log out with this link, and try to log in again, you will be asked to provide your username and password at Auth0 again.

Behind the scenes there are two places where this callback needs to be authorized (otherwise it won’t happen). First, vouch-proxy needs to find the url= parameter inside the list of post_logout_redirect_uris (check the config.yml). Second, the returnTo= parameter of the redirect needs to be added to Allowed Logout URLs in Auth0’s config. We have done this in the previous section. If something breaks for you, make sure to check these locations(and the returned X-Vouch-Error header).

Vouch-Proxy is working and communicating with Auth0! Next we will setup nginx as a reverse proxy sitting in front of vouch-proxy and our app.

Setup Nginx

The next step in the process is to setup an Nginx server that can act as a reverse proxy for our app and vouch-proxy. For this, create a new config file at ./nginx/conf.d/server.conf filled with the configuration below

server {
listen 80;
server_name localhost;
location ^~ /sso/ {
location /sso/validate {
proxy_pass http://vouch:9090/validate;
proxy_set_header Host $http_host;
proxy_pass_request_body off;
}
location = /sso/logout {
proxy_pass http://vouch:9090/logout?url=https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}%26returnTo=http://localhost/;
proxy_set_header Host $http_host;
}
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/;
}
# uncomment this to forward static content of vouch-proxy
# used when running vouch-proxy with `testing: true`
location /static/ {
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/static/;
}
location / {
root /usr/share/nginx/html;
index index.html;
}
}
view raw server.conf hosted with ❤ by GitHub
Initial nginx configuration.

Also, upate the docker-compose.yml like so

version: '3'
services:
vouch:
image: voucher/vouch-proxy
volumes:
– ./vouch/config/config.yml:/config/config.yml
nginx:
image: nginx
depends_on:
– vouch
volumes:
– ./nginx/conf.d/:/etc/nginx/conf.d/
ports:
– 80:80
view raw docker-compose.yml hosted with ❤ by GitHub
updated docker-compose.yml

and finally, update the vouch-proxy configuration to callback to the new location. For this, you only have to change the variable callback_url in the last line of the config file.

# vouch config
# bare minimum to get vouch running with OpenID Connect (such as okta)
vouch:
logLevel: debug
testing: true
listen: 0.0.0.0
port: 9090
allowAllUsers: true
jwt:
secret: Your-64-character-secret-key-here
issuer: Vouch
compress: false
cookie:
name: my-vouch-ct
secure: false
domain: localhost
headers:
jwt: X-Vouch-Token
querystring: access_token
redirect: X-Vouch-Requested-URI
accesstoken: X-Vouch-IdP-AccessToken
idtoken: X-Vouch-IdP-IdToken
post_logout_redirect_uris:
– https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}&returnTo=http://localhost/
oauth:
# Generic OpenID Connect
# including okta
provider: oidc
client_id: {client_id_from_auth0}
client_secret: {client_secret_from_auth0}
auth_url: https://your-project-name-here.eu.auth0.com/authorize
token_url: https://your-project-name-here.eu.auth0.com/oauth/token
user_info_url: https://your-project-name-here.eu.auth0.com/userinfo
scopes:
– openid
– email
– profile
callback_url: http://localhost/sso/auth
view raw config.yml hosted with ❤ by GitHub
updated vouch-proxy config.yml

Now update the docker stack so that it uses the new configuration.

What has just happened? We have added another node (nginx) to our docker stack and added a configuration for a server at the default http port. the first location block (^~ /sso/) acts as a reverse proxy for vouch-proxy. It has specializations for the /validate endpoint (no body needed), and for the /logout endpoint (for convenience). All authorization calls will, hence, go to localhost/sso/.

The second location block (/static/) handles requests to vouch-proxies static files (the logo, and .css you see for 302 calls). This block is only needed when we set testing: true in the vouch.proxy config. Otherwise, the debugging website will not be shown, and we can remove this block.

The third location block is where our app will live. For now, it is the default nginx website.

Let’s test this setup. First navigate to

localhost/

and make sure that nginx is up and running. Then navigate to 

localhost/sso/

and make sure that nginx is correctly forwarding to vouch-proxy (you should see the familiar 404 page not found). Now, test the login by navigating to

localhost/sso/login?url=http://localhost/sso/validate

This should result in the same flow that you are familiar with from the previous section, except that the URL now contains localhost/sso/ instead of localhost:9090/.

To log out simply visit

localhost/sso/logout

Notice how you are also logged out of Auth0. Nginx adds the necessary parameters before passing it to vouch proxy.

Secure your App

So far, we have setup nginx, vouch-proxy, and Auth0 in a neat docker stack and we have verified that everything is working. What we haven’t done yet is to integrate the actual app.

First, let’s create a super basic app that nginx can serve. Create a new file at ./web/index.html and fill it with a simple button to view protected content

<!DOCTYPE html>
<html>
<head>
<title>Simple App</title>
</head>
<body>
<button onclick="location.href='/sso/login?url=http\:\/\/localhost/protected';" id="myButton" class="float-left submit-button" >View Protected Content</button>
</body>
</html>
view raw index.html hosted with ❤ by GitHub
homepage of the app

and also a page that requires login to view at ./web/protected/index.html

<!DOCTYPE html>
<html>
<head>
<title>Simple App</title>
</head>
<body>
<p>This content is protected, and can't be seen without being logged in.</p>
<button onclick="location.href='/sso/logout';" id="myButton" class="float-left submit-button" >Logout</button>
</body>
</html>
view raw index.html hosted with ❤ by GitHub
protected page in the app

Then, add the files to the nginx container by updating the docker-compose.yml

version: '3'
services:
vouch:
image: voucher/vouch-proxy
volumes:
– ./vouch/config/config.yml:/config/config.yml
nginx:
image: nginx
depends_on:
– vouch
volumes:
– ./nginx/conf.d/:/etc/nginx/conf.d/
– ./web/:/usr/share/nginx/html/
ports:
– 80:80
view raw docker-compose.yml hosted with ❤ by GitHub
updated docker-compose.yml

Restart/update the stack, and you can see your website at localhost/. When clicking the “view protected content”, you will see the page that should be protected. When you click “logout” on the protected page, you will trigger the logout flow familiar from the previous sections.

To actually protect the content, we need to add a new location to nginx and protect it. This is done easily by updating the config file.

server {
listen 80;
server_name localhost;
location ^~ /sso/ {
location /sso/validate {
proxy_pass http://vouch:9090/validate;
proxy_set_header Host $http_host;
proxy_pass_request_body off;
}
location = /sso/logout {
proxy_pass http://vouch:9090/logout?url=https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}%26returnTo=http://localhost/;
proxy_set_header Host $http_host;
}
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/;
}
# uncomment this to forward static content of vouch-proxy
# used when running vouch-proxy with `testing: true`
location /static/ {
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/static/;
}
location / {
root /usr/share/nginx/html;
index index.html;
}
location /protected {
auth_request /sso/validate;
root /usr/share/nginx/html;
index index.html;
expires 0;
add_header Cache-Control "no-cache, no-store, must-revalidate, max-age=0";
add_header Pragma "no-cache";
}
}
view raw server.conf hosted with ❤ by GitHub
updated nginx server config

Now, when you click View Protected Content or manually navigate to localhost/protected, you will see the protected page (if you are logged in) or (if not) you will get a 401 Unauthorized error.

Next, we can have nginx catch the error and, instead of raising it, redirect the user to the login procedure with a simple addition to the server.conf

server {
listen 80;
server_name localhost;
location ^~ /sso/ {
location /sso/validate {
proxy_pass http://vouch:9090/validate;
proxy_set_header Host $http_host;
proxy_pass_request_body off;
}
location = /sso/logout {
proxy_pass http://vouch:9090/logout?url=https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}%26returnTo=http://localhost/;
proxy_set_header Host $http_host;
}
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/;
}
# uncomment this to forward static content of vouch-proxy
# used when running vouch-proxy with `testing: true`
location /static/ {
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/static/;
}
location / {
root /usr/share/nginx/html;
index index.html;
}
location /protected {
auth_request /sso/validate;
root /usr/share/nginx/html;
index index.html;
expires 0;
add_header Cache-Control "no-cache, no-store, must-revalidate, max-age=0";
add_header Pragma "no-cache";
error_page 401 = @prompt_login;
}
location @prompt_login {
return 302 http://localhost/sso/login?url=$scheme://$http_host$request_uri;
}
}
view raw server.conf hosted with ❤ by GitHub
updated nginx server config

Now, the user will be asked to log in if they try to access the protected location, and only if the login succeeds will they be able to view the protected page.

Bonus: Add HTTPS via self-signed certificates

In 2020 servers should enforce https, and while it is not necessary for localhost development, it is very nice to have. Especially later, when you have a development and production version of the code.

Adding SSH is very easy (shameless self plug). First generate a self-signed certificate for localhost (make sure to enter localhost as common name):

docker run --rm -it -v$PWD:/certs firefoxmetzger/create_localhost_ssl

(Source: https://github.com/FirefoxMetzger/create_ssl)

This will place a certificate and a private key into your current working directory which you can move to ./cert/ . Also, if you don’t want to be warned about an untrusted certificate, you can consider adding it to your browser’s trusted certificates.

Next, we have to update the server.conf for nginx

server {
listen 80;
server_name _;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name localhost;
ssl_certificate /certs/certificate.crt;
ssl_certificate_key /certs/private.key;
location ^~ /sso/ {
location /sso/validate {
proxy_pass http://vouch:9090/validate;
proxy_set_header Host $http_host;
proxy_pass_request_body off;
}
location = /sso/logout {
proxy_pass http://vouch:9090/logout?url=https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}%26returnTo=https://localhost/;
proxy_set_header Host $http_host;
}
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/;
}
# uncomment this to forward static content of vouch-proxy
# used when running vouch-proxy with `testing: true`
location /static/ {
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/static/;
}
location / {
root /usr/share/nginx/html;
index index.html;
}
location /protected {
auth_request /sso/validate;
root /usr/share/nginx/html;
index index.html;
expires 0;
add_header Cache-Control "no-cache, no-store, must-revalidate, max-age=0";
add_header Pragma "no-cache";
error_page 401 = @prompt_login;
}
location @prompt_login {
return 302 https://localhost/sso/login?url=$scheme://$http_host$request_uri;
}
}
view raw server.conf hosted with ❤ by GitHub
updated server.conf

The new first server block will forward all HTTP requests to HTTPS. Then, we add the SSL certificate we have just generated and change nginx to listen to the standard HTTPS port. Finally, we change the protocol from HTTP to HTTPS for both redirects.

Next update the callback_url for the vouch-proxy config as well as the post_logout_redirect_uris.

# vouch config
# bare minimum to get vouch running with OpenID Connect (such as okta)
vouch:
logLevel: debug
testing: true
listen: 0.0.0.0
port: 9090
allowAllUsers: true
jwt:
secret: Your-64-character-secret-key-here
issuer: Vouch
compress: false
cookie:
name: my-vouch-ct
secure: false
domain: localhost
headers:
jwt: X-Vouch-Token
querystring: access_token
redirect: X-Vouch-Requested-URI
accesstoken: X-Vouch-IdP-AccessToken
idtoken: X-Vouch-IdP-IdToken
post_logout_redirect_uris:
– https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}&returnTo=https://localhost/
oauth:
# Generic OpenID Connect
# including okta
provider: oidc
client_id: {client_id_from_auth0}
client_secret: {client_secret_from_auth0}
auth_url: https://your-project-name-here.eu.auth0.com/authorize
token_url: https://your-project-name-here.eu.auth0.com/oauth/token
user_info_url: https://your-project-name-here.eu.auth0.com/userinfo
scopes:
– openid
– email
– profile
callback_url: https://localhost/sso/auth
view raw config.yml hosted with ❤ by GitHub
updated conf.yml

Last, but not least, make the certificates available to nginx, by mounting the folder into the nginx container and open port 433 to allow SSL connections.

version: '3'
services:
vouch:
image: voucher/vouch-proxy
volumes:
– ./vouch/config/config.yml:/config/config.yml
nginx:
image: nginx
depends_on:
– vouch
volumes:
– ./nginx/conf.d/:/etc/nginx/conf.d/
– ./web/:/usr/share/nginx/html/
– ./cert/:/certs
ports:
– 443:443
– 80:80
view raw docker-compose.yml hosted with ❤ by GitHub
updated docker-compose.yml

Now, when you update the docker stack, you should be able to navigate to https://localhost (potentially receive a warning that the certificate could not be verified), and browse your app encrypted.

Remove Debugging

The only thing left is to remove some of the config that we have introduced for debugging purposes.

First, in the settings for the Tenant at Auth0, remove the uneeded allowed callback and logout URLs

Final Auth0 settings

Then, disable debug logs and testing mode in vouch-proxy.

# vouch config
# bare minimum to get vouch running with OpenID Connect (such as okta)
vouch:
listen: 0.0.0.0
port: 9090
allowAllUsers: true
jwt:
secret: Your-64-character-secret-key-here
issuer: Vouch
compress: false
cookie:
name: my-vouch-ct
secure: false
domain: localhost
headers:
jwt: X-Vouch-Token
querystring: access_token
redirect: X-Vouch-Requested-URI
accesstoken: X-Vouch-IdP-AccessToken
idtoken: X-Vouch-IdP-IdToken
post_logout_redirect_uris:
– https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}&returnTo=https://localhost/
oauth:
# Generic OpenID Connect
# including okta
provider: oidc
client_id: {client_id_from_auth0}
client_secret: {client_secret_from_auth0}
auth_url: https://your-project-name-here.eu.auth0.com/authorize
token_url: https://your-project-name-here.eu.auth0.com/oauth/token
user_info_url: https://your-project-name-here.eu.auth0.com/userinfo
scopes:
– openid
– email
– profile
callback_url: https://localhost/sso/auth
view raw config.yml hosted with ❤ by GitHub
final config.yml for vouch-proxy

And update the nginx config by removing the /static/ route.

server {
listen 80;
server_name _;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name localhost;
ssl_certificate /certs/certificate.crt;
ssl_certificate_key /certs/private.key;
location ^~ /sso/ {
location /sso/validate {
proxy_pass http://vouch:9090/validate;
proxy_set_header Host $http_host;
proxy_pass_request_body off;
}
location = /sso/logout {
proxy_pass http://vouch:9090/logout?url=https://your-project-name-here.eu.auth0.com/v2/logout?client_id={client_id_from_auth0}%26returnTo=https://localhost/;
proxy_set_header Host $http_host;
}
proxy_set_header Host $http_host;
proxy_pass http://vouch:9090/;
}
# uncomment this to forward static content of vouch-proxy
# used when running vouch-proxy with `testing: true`
#location /static/ {
# proxy_set_header Host $http_host;
# proxy_pass http://vouch:9090/static/;
#}
location / {
root /usr/share/nginx/html;
index index.html;
}
location /protected {
auth_request /sso/validate;
root /usr/share/nginx/html;
index index.html;
expires 0;
add_header Cache-Control "no-cache, no-store, must-revalidate, max-age=0";
add_header Pragma "no-cache";
error_page 401 = @prompt_login;
}
location @prompt_login {
return 302 https://localhost/sso/login?url=$scheme://$http_host$request_uri;
}
}
view raw server.conf hosted with ❤ by GitHub
final server.conf for nginx

Done! Now you have a simple app that is secured with Auth0 and vouch-proxy. You can add an API to this in the same way we have added the /protected route. Simply add 

auth_request /sso/validate;

to the route that proxies the API.

If you have any questions or comments, feel free to comment below.

Thanks for reading and Happy Coding!

Related Useful Links

Self-Signed SSL Certificate (https) for 127.0.0.1 (localhost)

I’ve updated my old repository on generating SSL certificates containing an IP SAN, which essentially allows to call https on IP addresses. It is still based on Docker, and now you can generate the certificates in a single command

docker run --rm -it -v$PWD:/certs firefoxmetzger/create_localhost_ssl

After filling in the required information, the container will generate the certificate, place it into your current folder and then self-destruct.

If you want the certificate stored in a different location, or if the $PWD environment variable isn’t defined (Windows, some unix variants). Replace $PWD with the location of your choice:

docker run --rm -it -v<absolute_path>:/certs firefoxmetzger/create_localhost_ssl

If you need to customize the certificate you can supply your own config file

docker run --rm -it -v$PWD:/certs -v<absolute/path/to/config.cfg>:/config.cfg firefoxmetzger/create_localhost_ssl

You can find the default config (and all other files) on GitHub, and – for your convenience – I pasted the config below, too.

[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
x509_extensions = v3_ca # The extentions to add to the self signed cert
string_mask = utf8only
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = AU
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Some-State
localityName = Locality Name (eg, city)
0.organizationName = Organization Name (eg, company)
0.organizationName_default = Internet Widgits Pty Ltd
organizationalUnitName = Organizational Unit Name (eg, section)
commonName = Common Name (e.g. server FQDN or YOUR name)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 64
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = CA:true
subjectAltName=@alternate_names
[ alternate_names ]
IP.1 = 127.0.0.1
view raw config.cfg hosted with ❤ by GitHub
Defaul config.cfg

Happy Coding.

Faster RNG for Reinforcement Learning in Python

When you start doing reinforcement learning you will sooner or later come to the point where you will generate random numbers. Initializing policy networks or Q-tables, choosing between exploration or exploitation, or selecting among equally good actions are a few examples. Numpy is very efficient at generating those random numbers, and most of the time (like 95%) this is all you need. However, there is one particular edge case were numpy is not the best solution, and that is exactly the case we encounter in RL a lot: generating single random numbers (i.e., to select an action epsilon-greedy).

Generating single random numbers in numpy is a bad idea, because every numpy call gets sent to the numpy engine and then back to python, which creates overhead that dominates runtime for single random numbers. In this case it is (much) more efficient to use the python standard library instead. However, if you can generate the random numbers in batches, numpy is significantly faster than the standard library again. Take a look at this simple profile:

import timeit
number = 10000
numpy_time = timeit.timeit("[np.random.rand() for _ in range(int(1e3))]", "import numpy as np", number=number)
random_time = timeit.timeit("[random.random() for _ in range(int(1e3))]", "import random", number=number)
numpy_batch_time = timeit.timeit("np.random.rand(int(1e3))", "import numpy as np", number=number)
print("Timings")
print("=======")
print(f"Numpy Single: {numpy_time:.3f}")
print(f"Random: {random_time:.3f}")
print(f"Numpy Batch: {numpy_batch_time:.3f}")
print("=======")
# =======
# Numpy Single: 3.245
# Random: 1.003
# Numpy Batch: 0.085
# =======
view raw random_profile.py hosted with ❤ by GitHub
Comparison between ways to generate random numbers

So if you do your profiling your code and notice that RNG adds up to a significant portion of your runtime, consider pre-generating the random numbers in numpy and then save them to a list. This solution sacrifices a bit of readability, but allows for much faster code. Here is an example that mimics the syntax of the python standard library:

import numpy as np
random_numbers = np.random.rand(int(2e8)).tolist()
def random():
try:
return random_numbers.pop()
except IndexError:
raise IndexError("Out of random numbers; generate more next time.")
def randint(a, b):
return int(round((b–a) * random())) + a
view raw RandomDropin.py hosted with ❤ by GitHub
Source Code for Random replacement.

That’s it. Thanks for reading and Happy Coding!

Parallelism in Python Generators

Yesterday I stumbled upon a StackOverflow question that asked about implementing a Rosetta Code problem in parallel to speed it up. One easy way to do it is one, which is a modification of the python example on rosettacode.org.

from multiprocessing import Pool, cpu_count
from itertools import islice, count
def is_special(n, d):
tofind = str(d) * d
return tofind in str(d * n ** d)
def superd(d, N=10000):
if d != int(d) or not 2 <= d <= 9:
raise ValueError("argument must be integer from 2 to 9 inclusive")
with Pool(cpu_count() – 2) as workers:
for offset in count(0, N):
worker_fn_args = zip(range(offset, offset + N), [d] * N)
is_superd_batch = workers.starmap(is_special, worker_fn_args)
yield from [n+offset for n in range(N) if is_superd_batch[n]]
if __name__ == '__main__':
for d in range(2, 10):
print(f"{d}:", ', '.join(str(n) for n in islice(superd(d), 10)))
view raw superd.py hosted with ❤ by GitHub
Source code for parallelism in generators

When I posted the question the OP commented that he/she was looking for using non-terminal sub-processes that yield these super-d values. I thought about it, but that version does not seem very practical. If the main process is interested in the results of the computation, then temporary concurrency will be the cleaner solution (like in this example). If the main thread isn’t, e.g., if you are running an old-school threaded web-server, that hands off incoming connections to a worker thread, then solutions with non-terminal sub-processes can make sense. In the latter case you are essentially “starting a program N times in parallel and shutting them down together”. This certainly makes sense, but only if those programs don’t need to communicate. Remember to KISS.

Thank you for reading and Happy Coding!

Franka Emika Panda in Gazebo with ROS and Docker

In my last post I’ve written about creating Gazebo packages from the existing panda models and mentioned that I’m also working on a docker image that includes ROS. Well here it is 🙂 You can check out the GitHub repository (be sure to leave a star if you like it), and – assuming you have an operating docker host – it should be very straight forward to set up.

Since I spent way too much time fixing dependencies and writing configuration files, I would like to share my experience (and of course the code :P) so that the solutions to the problems I encountered along the way are in one place.

High Level Overview

Before diving into the setup of the individual parts, I want to quickly talk about how the various components interact. This overview will make it easier to understand what each system is trying to accomplish, and what it expects.

A high level overview of how Panda is controlled in Gazebo. The simulator replaces the real robot at the level above the controllers (i.e. they use different controllers).

Contrary to my initially guess – that I could simply extend the Gazebo package I created earlier – I found that it is easier to start completely from scratch for this image. The logic here is that the robot will be spawned into Gazebo from ROS, and hence it makes sense to properly integrate things into the ROS architecture. This means using .urdf to describe the robot model, instead of .sdf files like I did in the previous post. It also means that I settled for one model – the robot with the gripper – although support for both versions can be added.

The main pipeline goes from MoveIt via ROS Control to Gazebo. MoveIt receives a goal pose and plans a trajectory to it. It then sends position commands to ROS control which sets targets for a series of PID controllers. These controllers are what steer the simulated joints in Gazebo. Gazebo then sends the joint encoder readings back to the ROS framework, where they are picked up by the PID controllers to stabilize the robot, MoveIt to decide the next step of the trajectory, and RViZ to visualize the robot. MoveIt and Gazebo both use the .urdf I generated in the last post, and I got the controller parameters from Erdal Pekel’s blog post.

ROS Control and Stabilizing the Robot

In retrospect getting the controllers to work is quite easy. While configuring it, it was quite annoying, because they didn’t seem to be loading properly. The issue was dependencies 🙂 Before I go into the packages needed to connect ROS control to gazebo, however, I want to show you the config file that I used to set the PID controls. I got this file from Erdal Pekel’s Blog who spend quite a bit of time tuning those numbers. Amazing job!

# panda: #useful if you use a namespace for the robot
# Publish joint states
joint_state_controller:
type: joint_state_controller/JointStateController
publish_rate: 50
panda_arm_controller:
type: effort_controllers/JointTrajectoryController
joints:
– panda_joint1
– panda_joint2
– panda_joint3
– panda_joint4
– panda_joint5
– panda_joint6
– panda_joint7
gains:
panda_joint1: { p: 12000, d: 50, i: 0.0, i_clamp: 10000 }
panda_joint2: { p: 30000, d: 100, i: 0.02, i_clamp: 10000 }
panda_joint3: { p: 18000, d: 50, i: 0.01, i_clamp: 1 }
panda_joint4: { p: 18000, d: 70, i: 0.01, i_clamp: 10000 }
panda_joint5: { p: 12000, d: 70, i: 0.01, i_clamp: 1 }
panda_joint6: { p: 7000, d: 50, i: 0.01, i_clamp: 1 }
panda_joint7: { p: 2000, d: 20, i: 0.0, i_clamp: 1 }
constraints:
goal_time: 2.0
state_publish_rate: 25
panda_hand_controller:
type: effort_controllers/JointTrajectoryController
joints:
– panda_finger_joint1
– panda_finger_joint2
gains:
panda_finger_joint1: { p: 5, d: 3.0, i: 0, i_clamp: 1 }
panda_finger_joint2: { p: 5, d: 1.0, i: 0, i_clamp: 1 }
state_publish_rate: 25
view raw panda_control.yaml hosted with ❤ by GitHub
PID parameters for the low-level joint controllers

To get these controllers to actually control the simulation, we can start them with a single line in the launch file, and point them to the `panda_controller.yaml`. To communicate with gazebo, however, they will need the `gazebo_ros_control` package, which doesn’t seem to ship with `gazebo_ros` nor with `ros_control`. Additionally the default ros container doesn’t ship with any controllers, so I had to install the effort controllers in the `effort_controllers` package, but also the `joint-trajectory-controller` which will install provide the controllers mentioned in above file. I didn’t test if the other effort controllers are necessary, so there might be a dependency on which this doesn’t actually depend.

If the low-level controllers are working properly, the arm in the simulation should freeze in a position that is different from the one it will settle in based on gravity. Usually, the simulation starts, and the arm falls. Then – a short while later – the controllers finish loading and stabilize the arm in some (slightly awkward) position.

Creating the MoveIt configuration

The MoveIt package in the repository (called `panda_moveit`) is the result of me using the moveit setup wizzard with the generated .urdf file. Initially, I tried using the official `panda_moveit_config` package, but failed to get things to work for two key reasons: (1) the official moveit config is meant for the physical robot, which uses controllers unsupported by Gazebo. This mismatch in controllers is detrimental. (2) I (unknowingly) chose a different naming convention in the .urdf compared to the officially used one. This can be fixed by me renaming things, but at this point I had already found out about reason (1), and thought I can keep it if I have to create my own config anyway.

To create this package, I followed the MoveIt Setup Assistant Tutorial, trying to stick as close to the existing `panda_moveit_config` as possible. For example, I included the ‘ready’ pose that panda uses to give the robot a default state 🙂

One file that I had to add manually to the generated configuration to get it to work was a config file for the MoveIt controllers. This tells MoveIT which controller groups it should use to actuate which motors, and what messages to send to the ROS control nodes. I then had to replace the `panda_moveit_controller_manager.launch.xml` with the newly created ones.

controller_list:
– name: panda_arm_controller
action_ns: follow_joint_trajectory
type: FollowJointTrajectory
joints:
– panda_joint1
– panda_joint2
– panda_joint3
– panda_joint4
– panda_joint5
– panda_joint6
– panda_joint7
– name: panda_hand_controller
action_ns: follow_joint_trajectory
type: FollowJointTrajectory
default: true
parallel: true
joints:
– panda_finger_joint1
– panda_finger_joint2
view raw panda_controllers.yaml hosted with ❤ by GitHub
The controller configuration for MoveIt

To start MoveIt and the planning context together with Gazebo, I added the `move_group.launch` file to the launch file in the `panda_gazebo` package. I didn’t touch the other launch files that the moveit setup assistant generated; hence, they are likely in a broken state.

Controlling the Simulator from RViZ

Finally, and to add some icing on the cake, controlling the Gazebo simulation through RViZ provides a very good test if everything is working as it should. I added the RViZ node to the launch file in the `panda_gazebo` package. After launching the file and waiting for everything to load, I created a layout in RViZ, configured it to my liking, and stored the layout as an .rviz config file. I then added this file as an argument to the launch options for the RViZ node.

A strange situation that I encountered here was that the static frame RViZ tries to use as reference in the global settings was set to `map` instead of `world`. I’m not 100% sure why the default is map, but if it is set to this, the interactive markers won’t show for panda’s endeffector.

With all this in place, I can now use the interact handles to drag the robot into a desired goal position and click ‘plan and execute’.

Putting It All Together in Docker

After all this work of assembling the pieces into a working pipeline, I thought I can take some additional time to alleviate some of the pain others may have when setting this up; especially the pain that comes from missing dependencies. As a result, I decided to create a docker image that is portable and will set all this up for you.

The Dockerfile itself is very short:

FROM osrf/ros:kinetic-desktop-full-xenial
RUN apt-get update \
&& apt-get install -y \
ros-kinetic-gazebo-ros \
ros-kinetic-gazebo-ros-pkgs \
ros-kinetic-gazebo-ros-control \
ros-kinetic-joint-state-controller \
ros-kinetic-effort-controllers \
ros-kinetic-position-controllers \
ros-kinetic-joint-trajectory-controller \
ros-kinetic-ros-control \
ros-kinetic-moveit\
&& rm -rf /var/lib/apt/lists/*
COPY assets/catkin_ws/src/panda_gazebo/models /root/.gazebo/models
COPY assets/catkin_ws/ /catkin_ws
COPY assets/ros_entrypoint.sh /ros_entrypoint.sh
CMD roslaunch panda_gazebo panda.launch
view raw Dockerfile hosted with ❤ by GitHub
Dockerfile used to generate the image

It just installs the necessary ROS packages (it is totally my own stupidity, but I can’t stress enough how much time I wasted figuring out which packages I need), adds the workspace created above, and then modifies the image’s entrypoint to lay the catkin workspace over the default ROS workspace.

One interesting thing happens in line 16, where I fix a bug that is very unique to gazebo in docker. As a container starts ‘fresh’ gazebo doesn’t have any models downloaded, which means it will download the two models it needs to create `empty.world`. This costs time and, unfortunately, interacts with ROS control in such a way that the controllers crash. To correct this, I pre-populate gazebo’s model cache with the models it needs.

The other file I use for building the image is the build script:

#!/bin/bash
# generate the .urdf
docker run –rm -v ${PWD}/assets/panda_xacro:/xacro osrf/ros:kinetic-desktop-full rosrun xacro xacro –inorder /xacro/panda_arm_hand.urdf.xacro > ${PWD}/assets/catkin_ws/src/panda_description/urdf/panda.urdf
# build the ROS packages
docker run –rm -it -v ${PWD}/assets/catkin_ws:/catkin_ws osrf/ros:kinetic-desktop-full-xenial bash -c "apt update && apt install -y python-catkin-tools && cd /catkin_ws && catkin build"
# Build the image
docker build -t panda_gazebo_sim:latest ${PWD}
view raw build.sh hosted with ❤ by GitHub
Build script for the docker image

The script is again very short. Line 4 generates the .urdf file from the .xacro file (a process I used in my post on how to get panda into Gazebo). Line 7 is interesting, because we spin up a clean ROS image to build the catkin workspace that we created, and then use those generated files to add them to the final image. Arguably this could be solved more elegantly via multi-stage builds; however, I learned about this feature after implementing things this way. Thus I will leave it as is until a future refactor, and will use multi-stage builds in future images.

That’s it! These posts usually take my entire weekend to write. If you think they are good, be sure to to like the post, and leave a star on my GitHub repository. This way I know that writing these is a worthy use of my time.

Happy coding!

H5-Index of HRI Venues

H5-Index of various venues typically targeted by researchers in HRI

I got bored during a late Saturday evening, so I decided to query Scopus for the last 5 years of publications for major venues in HRI. After aggregating all this data, I wanted to look at the impact factor and related metrics of these papers. However, I realized only afterwards that I can’t calculate the impact factor from the data I gathered. Instead I did the next best thing, computed the H5 scores for all the venues, and rank ordered them by this score. Maybe this graph is useful to some 🙂

Thank you for reading, and (although there is no programming here) happy coding!