Pages

2012-06-13

Motion Google Drive Uploader and Emailer

***** UPDATE: The script in this article no longer works due to Google authentication changes. Please use the new script described here: Motion Google Drive Uploader for OAuth 2.0



I'm using the brilliant Motion software with my Raspberry Pi to capture goings-on around my house. This blog post: Battery powered, Wireless, Motion detecting Raspberry Pi and this blog post: Raspberry Pi Webcam explain how.

I set up Lighttpd to point at the directory where Motion was creating video files so that I could go and check periodically to see if any new events had been captured. This works well - I just set up my home router so I could access the http over the Internet and then download the AVIs to my laptop or whatever. The down side to this is that you have to pro-actively look to see if there are new videos and secondly it puts additional strain on the Raspberry Pi to host the files for repeated download.

So, I have created a Python script to upload the AVI to Google Drive, get a URL to view the video on line and then email this using the same GMail account to an address. To keep the Raspberry Pi tidy the script then deletes the local file.

If you have the Google Drive Sync application running too you can see the files as they arrive which is fun:

Source


#!/usr/bin/python2
'''
Created on 6 Jun 2012

@author: Jeremy Blythe

Motion Uploader - uploads videos to Google Drive

Read the blog entry at http://jeremyblythe.blogspot.com for more information
'''

import smtplib
from datetime import datetime

import os.path
import sys

import gdata.data
import gdata.docs.data
import gdata.docs.client
import ConfigParser

class MotionUploader:
    def __init__(self, config_file_path):
        # Load config
        config = ConfigParser.ConfigParser()
        config.read(config_file_path)
        
        # GMail account credentials
        self.username = config.get('gmail', 'user')
        self.password = config.get('gmail', 'password')
        self.from_name = config.get('gmail', 'name')
        self.sender = config.get('gmail', 'sender')
        
        # Recipient email address (could be same as from_addr)
        self.recipient = config.get('gmail', 'recipient')        
        
        # Subject line for email
        self.subject = config.get('gmail', 'subject')
        
        # First line of email message
        self.message = config.get('gmail', 'message')
                
        # Folder (or collection) in Docs where you want the videos to go
        self.folder = config.get('docs', 'folder')
        
        # Options
        self.delete_after_upload = config.getboolean('options', 'delete-after-upload')
        self.send_email = config.getboolean('options', 'send-email')
        
        self._create_gdata_client()

    def _create_gdata_client(self):
        """Create a Documents List Client."""
        self.client = gdata.docs.client.DocsClient(source='motion_uploader')
        self.client.http_client.debug = False
        self.client.client_login(self.username, self.password, service=self.client.auth_service, source=self.client.source)
               
    def _get_folder_resource(self):
        """Find and return the resource whose title matches the given folder."""
        col = None
        for resource in self.client.GetAllResources(uri='/feeds/default/private/full/-/folder'):
            if resource.title.text == self.folder:
                col = resource
                break    
        return col
    
    def _send_email(self,msg):
        '''Send an email using the GMail account.'''
        senddate=datetime.strftime(datetime.now(), '%Y-%m-%d')
        m="Date: %s\r\nFrom: %s <%s>\r\nTo: %s\r\nSubject: %s\r\nX-Mailer: My-Mail\r\n\r\n" % (senddate, self.from_name, self.sender, self.recipient, self.subject)
        server = smtplib.SMTP('smtp.gmail.com:587')
        server.starttls()
        server.login(self.username, self.password)
        server.sendmail(self.sender, self.recipient, m+msg)
        server.quit()    

    def _upload(self, video_file_path, folder_resource):
        '''Upload the video and return the doc'''
        doc = gdata.docs.data.Resource(type='video', title=os.path.basename(video_file_path))
        media = gdata.data.MediaSource()
        media.SetFileHandle(video_file_path, 'video/avi')
        doc = self.client.CreateResource(doc, media=media, collection=folder_resource)
        return doc
    
    def upload_video(self, video_file_path):
        """Upload a video to the specified folder. Then optionally send an email and optionally delete the local file."""
        folder_resource = self._get_folder_resource()
        if not folder_resource:
            raise Exception('Could not find the %s folder' % self.folder)

        doc = self._upload(video_file_path, folder_resource)
                      
        if self.send_email:
            video_link = None
            for link in doc.link:
                if 'video.google.com' in link.href:
                    video_link = link.href
                    break
            # Send an email with the link if found
            msg = self.message
            if video_link:
                msg += '\n\n' + video_link                
            self._send_email(msg)    

        if self.delete_after_upload:
            os.remove(video_file_path)

if __name__ == '__main__':         
    try:
        if len(sys.argv) < 3:
            exit('Motion Uploader - uploads videos to Google Drive\n   by Jeremy Blythe (http://jeremyblythe.blogspot.com)\n\n   Usage: uploader.py {config-file-path} {video-file-path}')
        cfg_path = sys.argv[1]
        vid_path = sys.argv[2]    
        if not os.path.exists(cfg_path):
            exit('Config file does not exist [%s]' % cfg_path)    
        if not os.path.exists(vid_path):
            exit('Video file does not exist [%s]' % vid_path)    
        MotionUploader(cfg_path).upload_video(vid_path)        
    except gdata.client.BadAuthentication:
        exit('Invalid user credentials given.')
    except gdata.client.Error:
        exit('Login Error')
    except Exception as e:
        exit('Error: [%s]' % e)


Installation

If you haven't already got Python and Pip installed then do it now. On my Arch Linux Raspberry Pi I did this:
pacman -S python2
pacman -S python2-pip
Note: This script has been tested on Python 2.7.3
Next get the Google gdata library:
pip2 install gdata
Now download the two files above to somewhere on your machine. Make uploader.py executable:
chmod +x uploader.py
Open uploader.cfg to enter your settings:
[gmail]
# GMail account credentials
name = My Name
user = gmailusername
password = gmailpassword
sender = me@gmail.com

# Recipient email address (could be same as from_addr)
recipient = me@gmail.com

# Subject line for email
subject = Motion detected

# First line of email message
message = Video uploaded

[docs]
# Folder (or collection) in Docs where you want the videos to go
folder = motion

[options]
# Delete the local video file after the upload
delete-after-upload = true

# Send an email after the upload
send-email = true
Make sure you create a folder in your Google Drive to match the setting in the uploader.cfg file. This is the folder where the videos will go.
At this point it's worth testing the script e.g.
./uploader.py /etc/motion/uploader.cfg /srv/http/motion/cam1/85-20120608194940.avi
If all's well you can now change the Motion setting like so:
on_movie_end /root/py/uploader.py /etc/motion/uploader.cfg %f

Enjoy!

***** UPDATE: The script in this article no longer works due to Google authentication changes. Please use the new script described here: Motion Google Drive Uploader for OAuth 2.0


37 comments:

  1. Amazing articles! I have been following your installation of the webcam and this has been a great reference for info for the Raspberry Pi!

    ReplyDelete
  2. nice one! thanks a lot :)
    I try to make use of your script on an ubuntu box, but after installing python-gdata, i have an error on importing gdata.data?
    no module named data...
    I have to fix that

    ReplyDelete
  3. Fantastic stuff and worked a treat, I love it.

    Many thanks for the amount the work that's clearly gone into this and that you chose to share it.

    Very much appreciated

    ReplyDelete
  4. Jeremy, many thanks for putting these Pi tutorials together!

    I've got Motion up and running and am now trying to get the GDrive script to upload the video/images. When I run the script I get the generic "Login Error" and I'm not sure how to troubleshoot further.

    I've done a bit of troubleshooting up to this point, including pointing the script to a non-existent folder on gdrive, which led to me getting the "Could not find the %s folder" response. To me, this rules out a log-in problem.

    Any ideas?

    ReplyDelete
  5. @thehill06 looks like you might have to dig deeper into the gdata libraries to work out what's going on here - I wonder if there's more info behind that gdata.client.Error exception? In these situations I would pepper the code with print statements to see where it gets to before it falls over to hone in on the problem. Good Luck!

    ReplyDelete
  6. Got it working! This is brilliant. Thanks for your work and I look forward to seeing your other Pi projects.

    ReplyDelete
  7. @thehill06 Great! Check out my new post that hooks up LEDs and buttons to Motion: Raspberry Pi GPIO and Motion

    ReplyDelete
  8. @thehill06, I've got the same problem with "login error", however I'm not too sure where to start to debug the problem. What did you do to resolve the issue?

    ReplyDelete
  9. @ Xen, it turns out when I was doing my test upload I was trying to upload an image rather than a video file. Once I pointed the script to a video the issue was resolved.

    ReplyDelete
  10. Tanks for that excellent work! But i still have a litle problem...

    When i execute the python script, i get an error:

    Error: ['DocsClient' object has no attribute 'GetAllResources']

    Can anyone help me?

    Greetings from germany phate127

    ReplyDelete
  11. Does anyone have an example of how to upload an image instead of the avi to google.
    I've managed to get it to upload a jpeg by changing the doc type to image/jpeg, but viewing in google docs just opens up a blank document.

    ReplyDelete
  12. Hi Jeremy, thanks for the post - using this for my homebrew burglar detection system! A quick question - did you install a desktop environment and do your editing from a text editor or did you just use nano to edit the python scripts from the terminal?

    ReplyDelete
  13. @lewis I just use nano. I always connect in over ssh using PuTTY to do everything on the pi. It means less to install and more resources available. I also use svn to keep the code safe and then I can write it on my laptop and update on the pi.

    ReplyDelete
  14. Thanks for the quick response Jeremy - that clarifies it for me! Great tutorials.

    ReplyDelete
  15. Tanks for that excellent work! But i still have a litle problem...

    When i execute the python script, i get an error:

    Error: ['DocsClient' object has no attribute 'GetAllResources']

    Can anyone help me?

    Greetings from germany phate127

    Try using the later version of gdata. I got this error with 2.0.14, but 2.0.17 solved it.

    ReplyDelete
  16. Very nice work Jeremy. I have completed most of this setup, except the video upload. When testing the script as you describe it works well. However when adding into motion.conf I see that we are not specifying the full path and video name to upload. Filename I can understand, but where does your script get the path to where motion stores the video files?
    I see the variable "video_file_path" but dont see where it is defined.

    Many Thanks.

    ReplyDelete
  17. Jeremy, please ignore my previous comment, have fixed it - IT WORKS !!

    Many thanks for your efforts.

    ReplyDelete
  18. Hi Jeremy

    Just got Error: ['DocsClient' object has no attribute 'GetAllResources']

    ubuntu server 12.04 LTS
    python 2.7.3-0ubuntu2
    python-gdata 2.0.14-2

    any solution??

    thanks

    ReplyDelete
    Replies
    1. @cvaldess - I haven't seen that error before but I suspect it is something to do with the gdata install. You could try doing some other basic gdata things from the google tutorials to see if it's working.

      Delete
  19. Hi Jeremy,
    Thanks for the great post. I am got everything running but stuck on the last step. I can send force the upoloader to copy the video manually.

    But it does not work when the motion is running. It only uploads the vipdeo when I stop the motion by pressing control+c

    Any clues ?

    Thanks
    Mohsin

    ReplyDelete
  20. Also I only get the following trace when I press control+c after which it sends the file.

    Executing external command '/home/mohsin/Downloads/uploader.py /home/mohsin/Downloads/uploader.conf /tmp/motion/01-20120909040044.avi

    And I dont see this kind of trace while the motion is running.

    Thanks,
    Mohsin

    ReplyDelete
  21. Ignore my earlier comments. It does take a bit of time to send the message when the motion is running. It actually doe send the message.

    ReplyDelete
  22. Jeremy, run some examples from gdata-docs and work ok, think they don't support GetAllResources any more.. but not a python programer. thxn

    ReplyDelete
  23. Hi,

    Thanks for the script/blog post(s) - all very useful.

    Just wanted to highlight a minor-minor bug (sorry!) that might help other people...

    self.client.client_login(self.sender, ...

    should be

    self.client.client_login(self.username, ...

    That way it will log in with the username and not the sender e-mail address (in the case they are different) - that had me for a good while trying to figure out if Google's two-factor auth was to blame (I was using an app-specific password).


    Many thanks,
    Kenny

    ReplyDelete
  24. Hi, thank you for the great articles, I've successfully managed to get motion working on a fixed IP address and get to a point where i have the two scripts in directories on my RPi.

    I'm however having trouble with running and testing the 'uploader.py' script. I keep getting an error saying '-bash ./uploader.py: No such file or directory'

    I've only been playing with the Linux shell for a week and can't say I'm finding it as easy as I would have liked, so any help would be much appreciated!

    Thanks

    ReplyDelete
  25. Great script, Thanks!

    But could it be made to upload JPEG images instead? I've changed the MIME and type="images", however it seems Google tries to convert the image on upload. The URL needs to have '?convert=false' added at the end when uploading, but I can't figure out how. Any help?

    ReplyDelete
  26. All working very well. I modified to upload jpegs instead of avi's:

    doc = gdata.docs.data.Resource(type='image', title=os.path.basename(video_file_path))
    media = gdata.data.MediaSource()
    media.SetFileHandle(video_file_path, 'image/jpeg')


    One question though, I then thought I'd add the URL to the mail but the links don't work. They begin

    https://docs.google.com/feeds/default/contentmedia/folder

    and when accessed I get Invalid Request URI.

    Any ideas?

    ReplyDelete
  27. My jpegs are still coming in at 1kb, and as a .gdoc document. I can't figure out how to add the "?convert=false"

    ReplyDelete
  28. This comment has been removed by the author.

    ReplyDelete
  29. To add the ?convert=false while still posting to a collection, make @southscanner's changes, then
    replace

    doc = self.client.CreateResource(doc, media=media, collection=folder_resource)

    with

    create_uri = folder_resource.get_resumable_create_media_link().href + '?convert=false'
    doc = self.client.CreateResource(doc, create_uri=create_uri, media=media)

    See
    http://packages.python.org/gdata/docs/api.html
    --and--
    http://planzero.org/blog/2012/04/13/uploading_any_file_to_google_docs_with_python

    ReplyDelete
  30. Thanks for this script!

    The uploader works when running the script manually but not when the video has finished recording. I am using the exact same paths in both, except instead of ending with the filename in the config file I end with %f

    Any ideas?

    ReplyDelete
  31. Thanks for the scrip.I have the same problem as my previous poster, please help.

    ReplyDelete
  32. Not just for Pi lol. What a great post! Thanks Jeremy as this has helped me set up my motion setup with 4 webcams sending just the ones I need to Google Drive in photos. Great Post !

    ReplyDelete
  33. for wheezy:
    python not python2
    pip not pip2
    change first line of script (interpreter) to python2.7
    and works like a charm

    ReplyDelete
  34. I am using the ip camera to ftp the images to pi then upload it to drive. I can manually upload the images one by one but not when I use motion.cfg. I got both scripts and the jpgs are in one directory.
    SK

    ReplyDelete
  35. Hi, I have never used python or linux before and was wondering if you could explain exactly where to plug in the filenames/paths. I have tried a millions combos and am very frustrated.

    any help is greatly appreciated.

    ReplyDelete