Motion Google Drive Uploader for OAuth 2.0

Three years ago I wrote the original script to upload video to Google Drive as a trigger from Motion. Unfortunately Google recently removed support for the old security model and you now have to use OAuth 2.0. Also the gdata libraries have been superseded by the Google API Python Client. A few people have written alternatives to my original script quicker than I did, but here's my new version.

Note: I'm not using any third party libraries like PyDrive for example, this script goes straight to the Google APIs. (PyDrive has not been touched for a long time by its author and is missing features).

Motion Google Drive Uploader on Github

Installation

Google setup:

1. Go to https://code.google.com/apis/console - make sure you're logged in as the correct user for the Drive you want to use.
2. Use the drop-down at the top of the page to create a project called 'Uploader'.
3. Click on "Enable an API" then "Drive API" and then click the "Enable API" button
4. Next go to "APIs & auth -> Credentials" menu and click "Create new Client ID".
5. Set Application type to "Installed application" and click "Configure Consent Screen"
6. Set the Product Name to "Uploader"
7. You'll be returned back to the dialog. Make sure "Installed application" and "other" are selected and then click "Create Client ID".
8. Download the JSON file.
9. Rename it to client_secrets.json and put it somewhere like /home/pi/.

Script setup:

1. Go to /home/pi/ and get the uploader.py from github: git clone https://github.com/jerbly/motion-uploader.git
2. Update/install Google Python API: sudo pip install --upgrade google-api-python-client
3. Make uploader.py executable: chmod a+x uploader.py

Configuration:

1. If you used the git clone command you'll find the example uploader.cfg file in the motion-uploader directory. This now has a couple more options which need setting up. 
2. oauth/folder - set this to the directory where you have put the client_secrets.json file and where the script can write a credentials. e.g. /home/pi/
3. docs/snapshot-folder - set this to a public folder name in your drive if you wish to use this feature (explained below).

Initial authentication:

1. From the command line run the script by hand with your config file and a test avi file e.g. ./uploader.py /home/pi/uploader.cfg /home/pi/test.avi
2. Copy the URL to your clipboard and paste it into a browser where you are already logged in to Google as the correct user.
3. Accept and copy the authentication code to the clipboard.
4. Paste it back in to the command line and hit enter.
5. This will authenticate you and create a credentials file which will be used for future logins.

Now, finally, you can run up Motion and it should work like it used to.

New feature: Public Snapshot

Motion comes with a feature to periodically take a snapshot regardless of whether motion has been detected. This is a nice feature if you want to have a web site with the latest view from your webcam. You can use Google Drive to host this image rather than installing a web server on your Pi and opening firewalls etc.

1. Create a public folder in your Google Drive: 
1. Create a new folder called 'public'
2. Double click on it
3. Go to the sharing menu (person icon with + sign)
4. Click "Advanced" and then "Change..."
5. Set it to "On - Public on the Web"
2. Configure your uploader.cfg so docs/snapshot-folder is set to this folder 'public'.
3. Configure motion.conf to take a snapshot every n seconds named lastsnap.jpg and upload it:
1. snapshot_interval 300
2. snapshot_filename lastsnap
3. on_picture_save /home/pi/motion-uploader/uploader.py /home/pi/uploader.cfg %f snap

To find the public URL that you can bookmark and embed in other pages run the command line with the 'snapurl' option: ./uploader.py /home/pi/uploader.cfg /home/pi/lastsnap.jpg snapurl

It will print something like this: https://googledrive.com/host/{your-folder-id-here}/lastsnap.jpg

Raspberry Pi GPIO and Motion

This post shows how I've set up Motion with the Raspberry Pi GPIOs to indicate motion detection and video production on LEDs and a push button snapshot control. As part of this project I have created a Python service to allow easy scripted control of the GPIOs. This allows you to set up commands you want to run on GPIs and simple command line access to set GPOs high, low or flash. If you're new to these blog posts take a look at: Battery powered, Wireless, Motion detecting Raspberry Pi and Motion Google Drive Uploader and Emailer.

The electronics

I used the Electronic Starter Kit for Raspberry Pi plus a few extra jumper wires to give me 3 LEDs to control and two push button inputs:

The circuit uses the "low to glow" principle - set the GPO low to light the LED. I won't repeat a load of GPIO information available elsewhere, here's a very good article to read: Getting Started with Raspberry Pi GPIO and Python. I'm using 3.3v for the positive rail, pins 11, 12 and 13 for the red, yellow and green LEDs and 7 and 22 for the two buttons.

Flashing an LED from Motion

Motion has a number of configuration options which allow you to run shell commands when specific events occur. For example on_motion_detected, defined as Command to be executed when a motion frame is detected. The Motion web interface allows you to change all of these on the fly without restarting the service - great for testing. The on_motion_detected event does not have a sibling event like "on_motion_not_detected" so you can't easily switch an LED on on the first event and off on the second. Therefore you have to write a little bit of code to light the LED, wait for a short period and then switch it off. This is a common thing to want to do - a momentary indicator.

Another common thing to want to do is to flash an LED continuously, this gives you 3 indications for each LED: On, Off and Flash. But flashing can only be achieved by repeatedly setting the GPO high and low. I wanted to flash the red LED while Motion is in the "event window" this is defined by two events:

• on_event_start: Command to be executed when an event starts. An event starts at first motion detected after a period of no motion defined by gap.
• on_event_end: Command to be executed when an event ends after a period of no motion. The period of no motion is defined by option gap.

Taking a Motion snapshot

There are a few controls available for Motion through the action section of its web interface. The snapshot action captures a jpeg and creates a symlink to it called lastsnap.jpg. I then show this snapshot on my web site  (http://jerbly.uk.to/picam) next to the live feed. You can trigger a snapshot from the command line using curl like this:

curl http://localhost:8080/0/action/snapshot

All I needed was a way to run commands when a button is pressed. I needed to capture the GPI state going from False normally to True briefly (while the button is down) and then back to False again when it's released.

So, instead of writing more and more little scripts to handle controlling the GPIOs I wrote a service to do it for me. This allows me to echo simple instructions like "red flash" and "red high" to the service and it takes care of continuously flashing the GPO or setting it high etc.

My GPIO service

The code is available on GitHub here: Jerbly Raspberry Pi code. Download gpioservice.py and the example config file gpioservice.cfg to somewhere on your Raspberry Pi. Then, before you run it up, change the config file. This is very important as the config file determines which pins should be set as inputs and outputs and it also sets the initial mode of the pin (e.g. high). So you want to get this right before you start it up and blow something up!

The service gives you the follow features:

• Name to pin mapping: so you can send "red flash" or "motor low" commands
• Common modes: 
• High: set the pin high
• Low: set the pin low
• Flash: flip the pin between high and low every 0.25 second
• LowShot: set the pin low for 0.25s then leave it high
• HighShot: set the pin high for 0.25s then leave it low
• Commands are sent through a named pipe (fifo) which has write permissions for all, this means you don't have to be root to control the GPOs!
• Run shell commands for button presses on GPIs

The code in gpioservice.py uses two secondary threads alongside the main thread. The main thread is a blocking reader of the named pipe, it just parses the input on the pipe and sets the mode on the GPO objects. The GPO thread sleeps for 0.25s each loop, inverts the current flash state and then runs through each GPO object calling its action method. The action method determines, based on the mode, whether to set the pin high or low. It's this that means you can have flash, lowshot and highshot. The GPI thread sleeps for 0.1s each loop and then calls the action method on all the configured GPI objects. The action method reads the input and uses a state variable to trigger the command to run when the state first changes from False to True and not to continuously call the command if the button is held down. (Other modes may be added to this in the future so you can have "button held" operations).

Setting it all up

My Motion and GPIOs Raspberry Pi is set up to do this:

• Flash the red LED while in the "event window" (see above)
• Light the yellow LED when Motion starts making a video
• Light the green LED when it's finished making the movie and my Motion Google Drive Uploader and Emailer is running.
• When the upload has finished switch off the yellow and green LEDs
• When a Motion frame is detected briefly flash the green LED (lowshot)
• When the first button is pressed take a snapshot
• When the second button is pressed just lowshot the green LED for testing

Here's the gpioservice.cfg file:

[gpos]
fifo = /tmp/gpopipe

[gpo_pins]
# name = pin, initial_mode {high/low/flash}
red = 11, high
yellow = 12, high
green = 13, high

[gpi_pins]
# name = pin, command
motion_snapshot = 22, curl http://localhost:8080/0/action/snapshot
green_flash = 7, echo "green lowshot" >> /tmp/gpopipe

[options]
debug = False

Note that the fifo is configured at the top. The service takes care of creating this and setting the permissions. Then in the gpi_pins section you can see how to briefly flash the green LED when the button is pressed on pin 7.

Here's the Motion events section from my Motion config file:

# Command to be executed when an event starts. (default: none)
# An event starts at first motion detected after a period of no motion defined by gap
on_event_start echo "red flash" >> /tmp/gpopipe

# Command to be executed when an event ends after a period of no motion
# (default: none). The period of no motion is defined by option gap.
on_event_end echo "red high" >> /tmp/gpopipe

# Command to be executed when a picture (.ppm|.jpg) is saved (default: none)
# To give the filename as an argument to a command append it with %f
; on_picture_save value

# Command to be executed when a motion frame is detected (default: none)
on_motion_detected echo "green lowshot" >> /tmp/gpopipe

# Command to be executed when motion in a predefined area is detected
# Check option 'area_detect'. (default: none)
; on_area_detected value

# Command to be executed when a movie file (.mpg|.avi) is created. (default: none)
# To give the filename as an argument to a command append it with %f
on_movie_start echo "yellow low" >> /tmp/gpopipe

# Command to be executed when a movie file (.mpg|.avi) is closed. (default: none)
# To give the filename as an argument to a command append it with %f
on_movie_end /etc/motion/movie_end %f

# Command to be executed when a camera can't be opened or if it is lost
# NOTE: There is situations when motion doesn't detect a lost camera!
# It depends on the driver, some drivers don't detect a lost camera at all
# Some hang the motion thread. Some even hang the PC! (default: none)
; on_camera_lost value

Note that the on_movie_end setting calls a script that I have placed in /etc/motion to handle controlling the LEDs and uploading to Google Drive:

#!/bin/sh
echo "green low" >> /tmp/gpopipe
/root/py/jerbly/src/uploader.py /etc/motion/uploader.cfg $1
echo "green high" >> /tmp/gpopipe
echo "yellow high" >> /tmp/gpopipe

In Action

This first video shows the green lowshot by pressing the button connected to pin 7:

This second video shows the sequence when I trigger a motion event on the camera and start a recording. Note just after I remove the card from the camera there are two motion frame flashes on the green LED too.

Motion Google Drive Uploader and Emailer

***** UPDATE: The script in this article no longer works due to Google authentication changes. Please use the new script described here: Motion Google Drive Uploader for OAuth 2.0

I'm using the brilliant Motion software with my Raspberry Pi to capture goings-on around my house. This blog post: Battery powered, Wireless, Motion detecting Raspberry Pi and this blog post: Raspberry Pi Webcam explain how.

I set up Lighttpd to point at the directory where Motion was creating video files so that I could go and check periodically to see if any new events had been captured. This works well - I just set up my home router so I could access the http over the Internet and then download the AVIs to my laptop or whatever. The down side to this is that you have to pro-actively look to see if there are new videos and secondly it puts additional strain on the Raspberry Pi to host the files for repeated download.

So, I have created a Python script to upload the AVI to Google Drive, get a URL to view the video on line and then email this using the same GMail account to an address. To keep the Raspberry Pi tidy the script then deletes the local file.

If you have the Google Drive Sync application running too you can see the files as they arrive which is fun:

Source

#!/usr/bin/python2
'''
Created on 6 Jun 2012

@author: Jeremy Blythe

Motion Uploader - uploads videos to Google Drive

Read the blog entry at http://jeremyblythe.blogspot.com for more information
'''

import smtplib
from datetime import datetime

import os.path
import sys

import gdata.data
import gdata.docs.data
import gdata.docs.client
import ConfigParser

class MotionUploader:
    def __init__(self, config_file_path):
        # Load config
        config = ConfigParser.ConfigParser()
        config.read(config_file_path)
        
        # GMail account credentials
        self.username = config.get('gmail', 'user')
        self.password = config.get('gmail', 'password')
        self.from_name = config.get('gmail', 'name')
        self.sender = config.get('gmail', 'sender')
        
        # Recipient email address (could be same as from_addr)
        self.recipient = config.get('gmail', 'recipient')        
        
        # Subject line for email
        self.subject = config.get('gmail', 'subject')
        
        # First line of email message
        self.message = config.get('gmail', 'message')
                
        # Folder (or collection) in Docs where you want the videos to go
        self.folder = config.get('docs', 'folder')
        
        # Options
        self.delete_after_upload = config.getboolean('options', 'delete-after-upload')
        self.send_email = config.getboolean('options', 'send-email')
        
        self._create_gdata_client()

    def _create_gdata_client(self):
        """Create a Documents List Client."""
        self.client = gdata.docs.client.DocsClient(source='motion_uploader')
        self.client.http_client.debug = False
        self.client.client_login(self.username, self.password, service=self.client.auth_service, source=self.client.source)
               
    def _get_folder_resource(self):
        """Find and return the resource whose title matches the given folder."""
        col = None
        for resource in self.client.GetAllResources(uri='/feeds/default/private/full/-/folder'):
            if resource.title.text == self.folder:
                col = resource
                break    
        return col
    
    def _send_email(self,msg):
        '''Send an email using the GMail account.'''
        senddate=datetime.strftime(datetime.now(), '%Y-%m-%d')
        m="Date: %s\r\nFrom: %s <%s>\r\nTo: %s\r\nSubject: %s\r\nX-Mailer: My-Mail\r\n\r\n" % (senddate, self.from_name, self.sender, self.recipient, self.subject)
        server = smtplib.SMTP('smtp.gmail.com:587')
        server.starttls()
        server.login(self.username, self.password)
        server.sendmail(self.sender, self.recipient, m+msg)
        server.quit()    

    def _upload(self, video_file_path, folder_resource):
        '''Upload the video and return the doc'''
        doc = gdata.docs.data.Resource(type='video', title=os.path.basename(video_file_path))
        media = gdata.data.MediaSource()
        media.SetFileHandle(video_file_path, 'video/avi')
        doc = self.client.CreateResource(doc, media=media, collection=folder_resource)
        return doc
    
    def upload_video(self, video_file_path):
        """Upload a video to the specified folder. Then optionally send an email and optionally delete the local file."""
        folder_resource = self._get_folder_resource()
        if not folder_resource:
            raise Exception('Could not find the %s folder' % self.folder)

        doc = self._upload(video_file_path, folder_resource)
                      
        if self.send_email:
            video_link = None
            for link in doc.link:
                if 'video.google.com' in link.href:
                    video_link = link.href
                    break
            # Send an email with the link if found
            msg = self.message
            if video_link:
                msg += '\n\n' + video_link                
            self._send_email(msg)    

        if self.delete_after_upload:
            os.remove(video_file_path)

if __name__ == '__main__':         
    try:
        if len(sys.argv) < 3:
            exit('Motion Uploader - uploads videos to Google Drive\n   by Jeremy Blythe (http://jeremyblythe.blogspot.com)\n\n   Usage: uploader.py {config-file-path} {video-file-path}')
        cfg_path = sys.argv[1]
        vid_path = sys.argv[2]    
        if not os.path.exists(cfg_path):
            exit('Config file does not exist [%s]' % cfg_path)    
        if not os.path.exists(vid_path):
            exit('Video file does not exist [%s]' % vid_path)    
        MotionUploader(cfg_path).upload_video(vid_path)        
    except gdata.client.BadAuthentication:
        exit('Invalid user credentials given.')
    except gdata.client.Error:
        exit('Login Error')
    except Exception as e:
        exit('Error: [%s]' % e)

Installation

If you haven't already got Python and Pip installed then do it now. On my Arch Linux Raspberry Pi I did this:

pacman -S python2
pacman -S python2-pip

Note: This script has been tested on Python 2.7.3
Next get the Google gdata library:

pip2 install gdata

Now download the two files above to somewhere on your machine. Make uploader.py executable:

chmod +x uploader.py

Open uploader.cfg to enter your settings:

[gmail]
# GMail account credentials
name = My Name
user = gmailusername
password = gmailpassword
sender = me@gmail.com

# Recipient email address (could be same as from_addr)
recipient = me@gmail.com

# Subject line for email
subject = Motion detected

# First line of email message
message = Video uploaded

[docs]
# Folder (or collection) in Docs where you want the videos to go
folder = motion

[options]
# Delete the local video file after the upload
delete-after-upload = true

# Send an email after the upload
send-email = true

Make sure you create a folder in your Google Drive to match the setting in the uploader.cfg file. This is the folder where the videos will go.
At this point it's worth testing the script e.g.

./uploader.py /etc/motion/uploader.cfg /srv/http/motion/cam1/85-20120608194940.avi

If all's well you can now change the Motion setting like so:

on_movie_end /root/py/uploader.py /etc/motion/uploader.cfg %f

Enjoy!

***** UPDATE: The script in this article no longer works due to Google authentication changes. Please use the new script described here: Motion Google Drive Uploader for OAuth 2.0

Battery powered, Wireless, Motion detecting Raspberry Pi

Hardware

• Raspberry Pi
• ASUS USB-N10 Wireless Lan USB Adapter
• New Trent iCruiser 11000mAh External Battery Pack
• Microsoft Lifecam Cinema
• Waterproof plastic box

Software

• Arch Linux ARM
• Motion
• Lighttpd

Set up

Update Arch

Before installing any new software it's a good idea to get up-to-date. I was caught out with an issue which seems to be catching others out too - when upgrading I got this:

error: failed to commit transaction (conflicting files)
hwids: /usr/share/hwdata/pci.ids exists in filesystem
hwids: /usr/share/hwdata/usb.ids exists in filesystem
Errors occurred, no packages were upgraded.

In this case, for these two files, it's OK to rename them and try again - it seems that there's been a change of which package supplies these files and it's not handled very gracefully.

[root@alarmpi ~]# mv /usr/share/hwdata/pci.ids /usr/share/hwdata/pci.ids.save
[root@alarmpi ~]# mv /usr/share/hwdata/usb.ids /usr/share/hwdata/usb.ids.save
[root@alarmpi ~]# pacman -Syu

Wifi

I chose the ASUS USB-N10 because of its size and the fact that it works out-of-the-box with Arch. It uses the r8712u module which is already installed with the distro. Getting the wireless to work well enough is not particularly straightforward. However there are some good instructions here: ArchWiki Wireless Setup. For the wireless management software I chose netcfg and wrote a script similar to the one given under "Intermittent Connection Failure" so I had complete control over what I wanted to do whenever I lost connection to my router. This was helpful for troubleshooting and useful when you're walking around the garden trying to find a good place to put the box.

Motion

Motion is a great piece of software. Just use pacman to install it and then follow the instructions on the Motion site for setting this up. My top tip would be to edit /etc/motion/motion.conf to set up the stream and control ports and switch off webcam_localhost and control_localhost. Then run it on the console "motion -s" and use the web control to fiddle with the settings - for some changes you'll have to restart motion. Sometimes you might need a mask file. I found that taking one of the snapshots, loading it in Gimp and painting over the top was the best way to do this. You can then export the file in pgm format. At 320x240 the Raspberry Pi was quite happy to run at 5fps with motion using about 20 to 25% CPU and low memory usage (I've still got 194M free).

Lighttpd

To get at the snapshots and movies that Motion has captured it's nice to be able to grab them from a web page. I'm going to write some Python to do this nicely, but in the mean time I've set up lighttpd to present the capture directory really quickly. Simply use pacman to install it and then edit /etc/lighttpd/lighttpd.conf following the quick instructions here: Tutorial Configuration. 

Note: I am now uploading my videos to Google Drive rather than keeping them on the Raspberry Pi. See: Motion Google Drive Uploader and Emailer

Results

I have been consistently getting nearly 12 hours running time from the battery pack on a full charge. That gave me enough time to leave the box out all day and catch some "wildlife". Now I just need to write some software at last! (More posts on that coming soon)