Always Be Compacting

Running a big MongoDB installation requires a certain amount of routine maintenance. Over time, collections in a MongoDB database can become fragmented. This can be a particularly serious problem if your data usage patterns are relatively unstructured. In the long run, this can result in your databases taking up more space on disk and in RAM to hold the same amount of data, it can make many database operations noticeably slower, and it can reduce your overall query capacity significantly.

Conveniently, MongoDB provides 2 different ways to compact your data and restore optimal performance: repairDatabase and compact. RepairDatabase is appropriate if your databases are relatively small, or you can afford to take a node out of rotation for quite a long time. For our database sizes and query workload, it made more sense to run continuous compaction over all our collections.

To do this we wrote a small utility script to help us compact all our databases incrementally. We run this utility on a secondary node in our replicaset, and once it’s compacted everything, we can rotate that node in to be the primary node with minimal downtime. We also have this secondary node configured as our snapshot backup host, so if we ever need to reconstruct nodes from snapshots, the new nodes are as freshly compacted as possible.

Here’s how it works: it first fetches the list of all the databases in your replicaset, and then lists of all the collections in each database. It then goes through all of these collections and runs the compact command on each one. This is a blocking operation that puts the database into RECOVERY mode, so after each collection, it checks to see if replication has fallen too far behind, and if so, it waits for replication to catch up before resuming. If it’s interrupted or encounters an error, it saves the list of collections remaining to a file and then prints out instructions for how to resume it.

Here’s how to use it. To compact everything on the localhost mongo instance, you run it with no arguments (note that if you run this on the primary node, it will silently do nothing):

./mongo_compact.rb

To run it on a particular set of your databases (comma separated) you specify them with the -d option:

./mongo_compact.rb -d userdata1,userdata2,userdata3

To run it from cron, you use the -c option, and it will automatically save and resume its collection list in /var/run/mongo_compact/ and check if it’s already running using a .pid file in the same directory.

./mongo_compact.rb -c

You can see a full list of options with –help:

./mongo_compact.rb --help

You can find it in our public git repo for ops utilities: mongo_compact.rb

We hope you find this useful! If you want to hear more of these kind of tips, we’ll be sharing more of our tools and best practices at MongoSF on May 10th.

Brad Kittenbrink
March 26, 2013

Register for the Webcast: All About Parse Push Notifications

Parse Push

As one of the leading providers of push notifications, we get a lot of questions about our Parse Push offering. With the recent launches of Push Analytics and the Parse Push Console, we thought it was time to start educating our users on all the great things they can do with push notifications using Parse.

We’re excited to invite you to a webcast with Parse Solutions Architect, Héctor Ramos, and Senior Account Executive, Scott Smith covering the following topics:

  • What are push notifications?
  • What are some interesting ways successful applications are using push notifications?
  • Demo: Parse Push Console
  • Demo: Parse Analytics
  • Live Code Demo: Integrating push notifications into your iOS app

 

Click here for the re-cap of the “All About Parse Push Notifications” from April 10, 2013.

Ashley Smith
March 26, 2013

SpyThis! Puts a New Spin on an Old Classic

spythisThe proliferation of smart phones in today’s world presents the opportunity to put a new spin on classic games. Combining nostalgia with turn-based play can result in a highly-addictive gaming recipe, à la the popular “Words with Friends.”

We sat down with David Kennedy of Dangerous Pixels to learn more about their newest twist on a classic, SpyThis!

 

Tell us about your company and your role there.

I’m from Dangerous Pixels, which is a full service mobile studio located in Melbourne, Australia. We’re purely focused on designing and building mobile apps and have been doing so for a few years now. My main role is lead Interaction Designer, but I also co-own the business with Sam McCaig, who is the technical lead.

And what can you tell us about your new offering, SpyThis!?

Remember the classic game “I spy with my little eye …”? You probably played this in the car with your parents as a kid. Well, SpyThis! is a new twist on that. You can take a quick photo with your iPhone, create the game and send it to a friend to play.

People these days love sharing photos and interacting with each other via social media, and SpyThis! cleverly combines both of these into a unique social photo game. You can make the game as funny or as challenging as you want, or simply use it as a unique way of telling a friend what you’re up to. “I Spy” is no longer constrained to the car; you can play it anywhere, anytime, with your friends across the globe. It really is a lot of fun!

Where did the idea for the SpyThis! originate?

We were approached by Tom Advani and Max Towns, who together form the company Lazy Walrus, with the idea and concept for the app. They were looking for a local company to help them flesh it out and bring it to life. We loved the concept and it immediately sparked some great ideas of how we could do it, so it was an easy decision to partner with them.

What led you to Parse?

We have built many ‘backends’ before and know that the complexities in developing and maintaining them can quickly blow the budget for an app. We searched for a solution and came across Parse. After reading praise of the platform and giving it a bit of a test run, we decided to commit fully to using Parse.

Have you noticed any benefits to using Parse since you made the switch?

Because we are not spending hundreds of hours building reliable backend systems, we can spend more time on the core functionality of the app and create the best user experience possible. And, let’s be honest, Parse is cheap. Really cheap. Free, in fact, up to a point. How can you beat that? It gives indie developers a great platform to succeed on the app store with minimal upfront costs.

How is Parse used in the app?

In SpyThis!, Parse is used for everything: Push notifications, user management and authentication, social, in app purchasing, file storage and as a core data store.

Additionally, SpyThis! leverages the Facebook friends network. The login scheme for SpyThis! is Facebook only. To create a game in SpyThis!, you choose a Facebook friend, tag a photo, upload it, and send it off to your friend.  A push notification lets your friend know a game is there and waiting for you!

Has using Parse decreased development time and if so, do you have an idea of how much?

I would say Parse has saved us at least 50% of our time budget. Not only during development, but in maintenance efforts as well.

What do you love most about Parse?

What’s not to love? If I had to nail down one feature, it would be the user authentication schemes. The APIs just make it that much easier to integrate Facebook and Twitter into your apps.

 

SpyThis! is free and available for iPhone.

Courtney Witmer
March 25, 2013

Summary of the March 21st Parse Service Disruption

We would like to share more information about the service disruption on Thursday, March 21st, and the steps we are taking to prevent this sort of issue from happening again. We take reliability for our customers very seriously, and are always working to improve stability and uptime.

The outage began at 1:22 P.M. PDT, when all of the secondaries for one of our database clusters went into a crash loop. Upon investigation, we found that we had triggered a rare database bug where there was a corruption in the operation log used to synchronize from primary to secondary nodes. Our primary node was still alive, but we could not form a quorum to serve traffic from it. Unfortunately, our application routing logic had an incorrect dependency on the data stored in this database. This led to service disruption even for applications on separated clusters.

We were able to restore secondary nodes from snapshot to serve as arbiters and bring the cluster back up by 2:15 P.M., but we were then forced to snapshot off the primary to restore the cluster to a stable state. Service was intermittent for most users until approximately 5:00 P.M. The last few apps were restored to full functionality at 5:40 P.M.

In response to this incident, we have taken the following steps:

  • We have deployed a fix that will prevent us from triggering this database bug.
  • We are adding arbiters across multiple AWS availability zones so we do not need to rely on secondaries to elect a primary.
  • We are building increased isolation of services into our stack, so that one failing cluster will not affect our enterprise customers or customers hosted on other clusters.
  • We are developing custom database utilities that will allow us to manually skip or override corrupt operations.

We apologize for the outage. We know that platform downtime is highly disruptive to our customers. Our entire team is working very hard to increase reliability and prevent such an outage from happening again. If you have any questions, please reach out to us through our Help & Community portal.

Charity Majors
March 22, 2013

Stay Classy, Objective-C: Introducing Native Subclasses for Parse Objects

Thank you Developer! But our Business Logic is in Another Class

You wake up in a strange land. Somebody has stolen the princess and the town is looking to you to save her. Why don’t the townsfolk build their own army and fight the wizard themselves? Apparently they’re busy. For years, these townsfolk have perfected the art of Business Objects. They create two classes for every concept: one class contains business logic and the other is a SAX parser or some other delegate to manage storage and retrieval. These townsfolk aren’t warriors, they’re artisans. Each project requires new brilliant innovations in data mappings, and this has left them too busy to save the beloved princess.

“I need a weapon,” you say to yourself. Luckily, Parse has just the tool for you.

It’s Dangerous To Go Alone! Take This

Objective-C is a dynamic language, and dynamic languages offer interesting possibilities. Today we are releasing support for native subclasses in Objective-C. With proper subclassing, your PFObjects are your business objects. Your subclasses can have the properties and methods that are unique to a particular business function. Your code becomes terser, easier to read, and supports autocomplete in Xcode. Imagine turning the following code:

PFObject *shield = [PFObject objectWithClassName:@"Armor"];
[shield setObject:@"Wooden Shield" forKey:@"displayName"];
[shield setObject:[NSNumber numberWithBool:NO] forKey:@"fireproof"];
[object setObject:[NSNumber numberWithInt:50] forKey:@"rupees"];

into this:

Armor *shield = [Armor object];
shield.displayName = @"Wooden Shield";
shield.fireproof = NO;
shield.rupees = 50;

It’s Super Effective!

It’s easy to set up PFObject subclasses, and it greatly simplifies app development. There are built-in helpers for creating and querying for objects. You can better encapsulate logic for a particular class by adding custom properties and methods. Consider the complete implementation of the armor class:

Armor.h

@interface Armor : PFObject<PFSubclassing>
+ (NSString *)parseClassName;
@property (retain) NSString *displayName;
@property int rupees;
@property BOOL fireproof;
@end

Armor.m

#import "Armor.h"
#import <Parse/PFObject+Subclass.h>

@implementation Armor 
@dynamic displayName;
@dynamic rupees;
@dynamic fireproof;
+ (NSString *)parseClassName {
  return @"Armor";
}
@end

We can now create objects using [Armor object] and query for Armor objects with [Armor query]. The definition of parseClassName ensures that [PFObject objectWithClassName:@"Armor"] will create an Armor object. PFObject supports dynamic synthesizers; anArmor.displayName is now equivalent to [anArmor objectForKey:@"displayName"] but is terser, supports code complete, and is a compiler error when mistyped. As an NSString, we can even declare displayName to be a copy property as well.

The rupees and fireproof properties here are extra special: PFObjects require object members, but the rupees property is an int and fireproof is a BOOL. PFObject properties support unboxing. [anArmor objectForKey:@"fireproof"] returns an NSNumber with a boolValue, but setting the fireproof property automatically wraps the BOOL in an NSNumber, and getting the fireproof property automatically extracts the boolValue from the NSNumber.

(Nearly) All Your Implementation Are Belong To Base Class

This change is made possible through great new features in PFObject which enable subclassing. To keep code clean, the methods which only make sense in subclasses (e.g. initializers without a class name) are exposed in the new PFSubclassing protocol. Everything but the name of the class you want to implement are already implemented in the PFObject Subclass category, so setup is simple. Read more about our setup in the iOS/OS X guide.

Good luck saving the princess!

Thomas Bouldin
March 22, 2013

Parse Sponsors Kiip’s Build Fund 2.0 for Independent App Developers

Kiip

Today we’re proud to announce our sponsorship of the Kiip 2013 “Creation Build Fund for Independent App Developers. Open to all independent app developers, this year’s ten winners will receive $10,000 in cash and $5,000 worth of services from Kiip’s chosen partners, including 6 months of free Parse Pro.

About Kiip 

Kiip is the world’s first mobile rewards network that delivers rewards for achievements in apps and games. The company’s category-creating rewards platform enables brands to reach consumers in the moments when they are most engaged and receptive, while driving revenues and greater user allegiance for Kiip-enabled games and apps. Kiip was founded in 2010 by Brian Wong, Courtney Guertin and Amadeus Demarzi. In addition to its San Francisco headquarters, Kiip now has offices in New York City, Los Angeles, Chicago, London and Bogotá. Kiip is currently backed by Relay Ventures, IPG, Hummer Winblad, True Ventures, Digital Garage, Verizon Ventures, Crosslink Capital and others. For more information, visit www.kiip.com.

Ashley Smith
March 22, 2013

Build an e-commerce mobile app on Parse and Stripe

parse_store

Many of our developers want to accept payments in their mobile apps so that they can build experiences like Amazon, Zappos, Lyft, and Uber. Today, we are happy to announce a partnership with Stripe that enables these kinds of apps.

Stripe is an online payment processing platform that is renowned for its simplicity and transparency. Over the last few years, Stripe has gained great popularity and adoption in the web developer community. The integration with Parse allows mobile developers to easily take advantage of Stripe’s payment processing platform.

No Server Needed

In the past, to charge a credit card using Stripe, developers needed to set up an independent server to validate each purchase. With Parse, developers no longer need to manage their own servers. Using the Stripe Cloud Module, developers can validate the Stripe tokens on the Parse cloud. The following diagram demonstrates this interaction at work:
parse_stripe_diagram 2

Parse Store

We built a sample app to demonstrate what you can do with the Parse + Stripe integration. Parse Store is an iOS app that lets you buy Parse-branded hoodies, T-shirts, and mugs. This app is also available on the App Store for you to play with. Give it a shot! We’ll send you the Parse-branded product of your choice!

parse_store_1
parse_store_2
parse_store_3

We have open-sourced the Parse Store app so that developers can look at the source code to understand how to use Parse with Stripe. Developers who aspire to set up a mobile e-commerce store are also welcome to jumpstart their development using the app.

Let us know what you think; we love to hear your feedback!

Andrew Wang
March 21, 2013

Partnering with New Relic for Performance Management

We see a wide variety of customers here at Parse, from indie developers making games like Hotel Story, to massive household names building apps to support their brand, such as the Food Network. At every level, though, when a developer is considering a service provider, their concerns remain constant: Is this reliable? Will my data be secure? Am I going to have to migrate my data if we get popular, or is this a scalable solution? Does this offer all of the tools I’m looking for in an intuitive, user-friendly way? Parse is here to make sure that when you decide to use us to support your backend, the answers to all of those questions is “yes.”

Today’s developers are looking for one-stop solutions; no one wants to manage multiple accounts across different platforms or use different providers for the various features in an app, like push and data. We’re seeing an increase in MBaaS providers as a result, and Parse is proud to be the clear leader in the space. Mobile developers want to create apps that are not only useful and engaging but also fast and reliable. That’s why we’re constantly working to provide the features our customers need and want, and this partnership with New Relic is yet another example of working to bring our developers the tools they need to create engaging, dynamic apps that are also reliable and scalable.

New Relic’s performance management capabilities for native mobile apps provide the visibility developers need to ensure their apps are rock solid. Together, Parse and New Relic allow developers to focus on creating the very best apps for their users with the confidence that critical performance data is always at their fingertips.

Check out this video where I discuss the mobile market, our partnership with New Relic, and the value that New Relic for Mobile apps brings to our customers.

Getting Started with New Relic for Mobile Apps

Through this unique partnership, Parse customers get New Relic Standard free of charge, including web monitoring and mobile monitoring for unlimited apps and users. Parse customers can sign up for their free New Relic Standard account today!

Ilya Sukhar
March 20, 2013

Webcast Recap: Developing Windows 8 and Windows Phone 8 Apps Using Parse

Thanks to everyone who attended today’s webcast, “Developing Windows 8 and Windows Phone 8 Apps Using Parse,” with David Poll, Software Engineer at Parse, and JC Cimetiere, Director for Windows Phone Partner and Developer Programs, Microsoft. For anyone who would like a second look, or just in case you missed it, the full video is below.

The sample code David reviewed can be found here. If you’d like to book office hours with Matt Harrington, Developer Evangelist for Microsoft, to learn more about Windows Phone 8 or Windows 8 here.

Don’t forget to register for the next session of the Parse and Windows Phone and Windows Phone 8 Web Series here for our next webcast, “Cross Platform Development for Windows Apps Using Parse.”

Watch the recording:

Ashley Smith
March 20, 2013

Implementing Scalable Search on a NoSQL Backend

Search is really easy to implement for a quick and dirty prototype, but really hard to implement in a scalable way for production usage. The patterns that work most conveniently for prototyping are often the direct cause of scalability problems later in an application’s life cycle.

Simplistic search algorithms simply scan through all the documents and execute the query on each one. If it sounds like this can take a long time, that’s because it does. The key to making searches run efficiently is to minimize the number of documents that have to be examined when executing each query by using an index. To do that you need to keep in mind what kinds of queries you want to support when designing how to organize your data. The more structured and limited these queries are, the easier this will be.

Dont Worry We Got This Cats
One of the big advantages of using a service like Parse is that you don’t have to worry about managing your own database and maintaining indexes. We’ve built an abstraction for you so that we manage all of that complexity for you, and you can focus on your application’s unique features.

To organize your data model to support efficient searching, you’ll need to know a bit about how our systems are operating behind the abstraction. You’ll need to build your data model in a way that it’s easy for us to build an index for the data you want to be searchable. For example, string matching queries that don’t match an exact prefix of the string won’t be able to use an index. This makes these types of queries very likely to fail due to timeout errors as your app grows.

We’ve been recently adding features that make it easier to search your data efficiently, and we want to continue making that easier for our users. Let’s look at an example: Say your app has users making posts, and you want to be able to search those posts for hashtags or particular keywords. You’ll want to pre-process your posts and save the list of hashtags and words into array fields. You can do this processing either in your app before saving the posts, or you can just add a Cloud Code hook to do it on the fly, leaving your app code unchanged.

Here’s an example Cloud Code hook to do this for posts:

var _ = require("underscore");
Parse.Cloud.beforeSave("Post", function(request, response) {
    var post = request.object;

    var toLowerCase = function(w) { return w.toLowerCase(); };

    var words = post.get("text").split(/\b/);
    words = _.map(words, toLowerCase);
    var stopWords = ["the", "in", "and"]
    words = _.filter(words, function(w) { return w.match(/^\w+$/) && ! _.contains(stopWords, w); });

    var hashtags = post.get("text").match(/#.+?\b/g);
    hashtags = _.map(hashtags, toLowerCase);

    post.set("words", words);
    post.set("hashtags", hashtags);
    response.success();
});

This saves your words and hashtags in array fields, which MongoDB will store with a multi-key index. There are some important things to notice about this. First of all it’s converting all words to lower case so that we can look them up with lower case queries, and get case insensitive matching. Secondly, it’s filtering out common words like ‘the’, ‘in’, and ‘and’ which will occur in a lot of posts, to additionally reduce useless scanning of the index when executing the queries. Long story short, that means you can efficiently look them up using All queries.

For example, in iOS or OS X:

PFQuery *query = [PFQuery queryWithClassName:@"Post"]
[query whereKey:@"hashtags" containsAllObjectsInArray:@[@"#parse", @"#ftw"]];
NSArray *parseFTWPosts = [query findObjects];

Or using our REST API:

curl -v -X GET  \
    -H "X-Parse-Application-Id: ${APPLICATION_ID}" \
    -H "X-Parse-REST-API-Key: ${REST_API_KEY}" \
    -G \
    --data-urlencode 'where={"hashtags":{"$all":["#parse", "#ftw"]}}' \
    "https://api.parse.com/1/classes/Post"

Give it a go. Implementing search using these patterns will help make your apps run faster.

Brad Kittenbrink
March 19, 2013

Archives

Categories

RSS Feed Follow us Like us