Google App Engine Image Storage?

My App Engine project: PicSoup, is a weekly photo voting competition with a twist (read the Help tab at the site for instructions). The app is all about images. Users upload (or email in) their competition entry which is then resized and displayed on the site for everyone to vote for.

The Blobstore is used for the initial upload to get around the 1MB limit. Sadly this is not available for emails so the user is limited to 1MB for email entry. The large image is then resized and stored as a blob in the Datastore like this:

Map<String, BlobKey> blobs = blobstoreService.getUploadedBlobs(req);
BlobKey blobKey = blobs.get("upload");

// Transform the image to a small one and save it in the datastore
ImagesService imagesService = ImagesServiceFactory.getImagesService();

Image oldImage = ImagesServiceFactory.makeImageFromBlob(blobKey);

Transform resize = ImagesServiceFactory.makeResize(250, 220);

Image newImage = imagesService.applyTransform(resize, oldImage, ImagesService.OutputEncoding.JPEG);
   
Pic pic = new Pic();
pic.setImage(new Blob(newImage.getImageData()));           
dao.putPic(pic);

When it's time to serve an image the servlet code is pretty simple.

public void doGet(HttpServletRequest req, HttpServletResponse res) throws IOException {
    String picId = req.getParameter("id");
    if (picId != null) {
        Pic p = dao.getPic(Long.parseLong(picId));
        if (p != null) {
            res.setContentType("image/jpeg");
            res.setHeader("Cache-Control", "max-age=1209600"); //Cache for two weeks
            res.getOutputStream().write(p.getImage().getBytes());                
        }
    }
}

Note that I'm using the Cache-Control header with a max-age of two weeks. The image with the given id never changes so in theory this could be set to forever. This caching is very important because otherwise the app gets hit every time for the image. Users of PicSoup frequently visit the site to check for new entries .

The downside to this is that sometimes the Datastore can be very slow. I've watched images appear like they used to on an old 56k modem! Google were having some problems with Datastore performance and it is way better now but it's not as fast as accessing a static file on a dedicated server.

The performance and the Datastore usage quota put me off keeping a higher resolution image but the site really needed it. So I started developing a way to store the big images in Picasa. The documentation is really good and I soon had this working. Now when the user uploaded their image the small image would still be stored in the Datastore as above but then a task would be enqueued on the Task Queue to transform the image from the Blobstore again and then upload it to my Picasa account:

private void addToPicasa(String blobKey, String id) {
    ImagesService imagesService = ImagesServiceFactory.getImagesService();
    BlobKey bk = new BlobKey(blobKey);
    Image oldImage = ImagesServiceFactory.makeImageFromBlob(bk);
    Transform resize = ImagesServiceFactory.makeResize(1024, 768);
    Image image = imagesService.applyTransform(resize, oldImage, ImagesService.OutputEncoding.JPEG);
    logger.info("Big image bytes: "+image.getImageData().length);
            
    PicasawebService myService = new PicasawebService("PicSoup");
    try {
        myService.setUserCredentials("mypicasaaccount@gmail.com", "my.password");
        String albumid = "5495486978942507441";
        if (SystemProperty.environment.value() == SystemProperty.Environment.Value.Production) {
            albumid = "5495487073314754801";
        }
        URL albumPostUrl = new URL("http://picasaweb.google.com/data/feed/api/user/mypicasaaccount/albumid/"+albumid);

        PhotoEntry myPhoto = new PhotoEntry();
        myPhoto.setTitle(new PlainTextConstruct(id));

        MediaSource myMedia = new MediaByteArraySource(image.getImageData(), "image/jpeg");
        myPhoto.setMediaSource(myMedia);

        PhotoEntry returnedPhoto = myService.insert(albumPostUrl, myPhoto);            
        MediaContent mc = (MediaContent) returnedPhoto.getContent();

        CompEntry ce = dao.ofy().get(CompEntry.class, Long.parseLong(id));
        ce.setPicUrl(mc.getUri());
        ce.setPhotoId(returnedPhoto.getGphotoId());
        
        dao.ofy().put(ce);
    } catch (Exception e) {
        logger.error(e);
    }
    
    // Delete the hi-res
    blobstoreService.delete(bk);        
}

Before I started the UI work to display the image from Picasa I checked the Terms of Service and realised this solution may be contrary to item 5.9:

5.9 In order to use the Picasa Web Albums API with your service, all End Users on your service must have previously created their own individual Picasa Web Albums accounts. You must explicitly notify End Users that they are accessing their Picasa Web Albums accounts through your service. In other words, you may not create one or more Picasa Web Albums accounts for the purpose of storing images on behalf of users without those users creating their own individual Picasa Web Albums accounts.

Now, strictly, I'm not sure I'm storing the images on behalf of the users - they've kind of donated them to me and my app. I searched around for some clarification and found that there are plenty of people trying to do this sort of thing and the answer is always no. Have a look at this search in the forum.

So, I've removed Picasa from my app and I'm now using the Datastore to hold an 800x600 image as well. (If you go to PicSoup today [31-July-2010] only the most recent entries have the high-res view available, just click the small image). Now that the Datastore performance has improved this is not so bad.

I've looked at Flickr and Photobucket as well and they also seem to have a clause like this in their terms.

Does anyone know of a service where this is allowed?

UPDATE See my new post which explains how to use the new Blobstore based fast image serving service.