Parse Command Line Gets a Power Up

As more people use Cloud Code, we strive to improve the tool that directly supports it, the Parse Command Line Tool. Today, we’re announcing three new features we’ve added to our command line tool: parse updateparse default, and parse jssdk.

parse update

As the name implies, this command helps you update the tool to its latest version. If you’ve previously updated the tool, you had to revisit the Parse docs and copy/paste this script to your terminal:

curl -s https://www.parse.com/downloads/cloud_code/installer.sh | sudo /bin/bash

This is where parse update comes to the rescue! Now, you can update directly from version 1.3.0 or later. This feature is currently only available for Unix-based operating systems, but we’re hard at work getting it added to Windows.

parse default

This command returns the default app that will be targeted with our existing features, such as parse deploy, parse log, parse releases and parse rollback, when an app is not explicitly provided. Before this patch, you had to edit your config file and the _default‘s link to your desired app if you wanted to leverage the default app behavior. Now, you can use parse default [app name] to set your default app.

parse jssdk

This feature returns the current Parse JavaScript SDK used on your Cloud Code. You can also explicitly set your current JavaScript SDK by using parse jssdk [version]. This allows you to have control over the Parse JavaScript SDK on your Cloud Code. You can see all available Parse JavaScript SDK versions by running parse jssdk -a.

$ parse jssdk -a
  1.2.17
  1.2.16
  1.2.15
  1.2.14
* 1.2.13
  1.2.12
  1.2.11
  ...

You’ll find the full docs here. Now, go update your command line tool and let us know what you think!

Listiarso Wastuargo
April 9, 2014

A Toast to Windows Phone Users

Did you know that our Push Notifications product supports Windows devices? As developers, we know how important it is to maximize the ability to reach out to your users, regardless of which phone they might be using. For a rapidly growing subset of mobile phone owners, those devices are running Windows, and we wanted to make sure that sending notifications to those users was as easy and powerful as our existing offerings for iOS and Android.

With Windows Phone on our list of supported platforms, you can leverage our API to gain thorough control of when and where you send Windows Toast Notifications. Of course, if you’ve used us for Push before, you know that the simplest method for scheduling notifications has always been through our Push Console – don’t worry, you’ll find an option for Windows Phone there as well.

Windows Phone Push Console

You can configure your Windows Phone app to accept Push Notifications by following the instructions in our Docs. Once you have registered installations of your app, you’ll be able to select the Windows Phone checkbox in the Push Console and see a preview of the Toast Notification that will appear on your users’ devices. Selecting the times and recipients of your notifications is as easy as it always was with the Push Console – the query builder allows you to choose specific subsets of your users, making sure your message gets to exactly the right people.

As always, you can read about all of our Push Notification services in our Docs, including how to track app opens through notifications.

Andrew Imm
April 3, 2014

Smart Indexing at Parse

We love running databases at Parse! We also believe that mobile app developers shouldn’t have to be DBAs to build beautiful, performant apps.

No matter what your data model looks like, one of the most important things to get right is your indexes. If your data is not indexed at all, every query will have to perform a full table scan to return even a single document. But if your data is indexed appropriately, the number of documents scanned to return a correct query result should be low.

We host over 200k mobile apps at Parse, so obviously we can not sit down and individually determine the correct indexes for each app. Even if we did, this would be an ineffective mechanism for generating indexes because developers can change their app schemas or query access patterns at any time. So instead we rely on our algorithmically generated smart indexing strategy.

We perform two types of index creation logic. The first generates simple indexes for each API request, and the second does offline processing to pick good compound indexes based on real API traffic patterns. In this blog post I will cover the strategies we use to determine simple single-column indexes. We will detail our compound indexing strategies in a followup post.

At a high level, our indexing strategy attempts to pick good indexes based on a) the order of operators’ usefulness and b) the expected entropy of the value space for the key. We believe that every query should have an index; we choose to err on the side of too many indexes rather than too few. (Every index causes an additional write, but this is a problem that is easier to solve operationally than unindexed queries.)

Operator usefulness is determined primarily based on its effectiveness in providing the smallest search space for the query. The order of operators’ usefulness is:

  • equals
  • <, <=, =>, >
  • prefix string matches
  • $in
  • $ne
  • $nin
  • everything else

For data types, we heavily demote booleans, which have very low entropy and are not useful to index. We also throw out relations/join tables, since no values are stored for these keys. We heavily promote geopoints since MongoDB won’t run a geo query without a geo index. Other data types are ranked by their expected entropy of the value space for the key. Data types in order of usefulness are:

  • array
  • pointers
  • date
  • string
  • number
  • other

We score each query according to the above metrics, and run ensureIndex() on the three top-scoring fields for each query. If the query includes a $or, we compute the top three indexes for each branch of the $or.

When we first started using smart indexing, all indexes were ensured inline with each API request. This created some unfortunate pile-on behavior any time a large app had schema or query pattern changes. Instead of ensuring the index in the request path, we now drop the indexing job in a queue where it is consumed by an indexer worker. This ensures that no more than one index can be generated at a time per replica set, which protects us from these pile-on effects.

Even the best indexing strategy can be defeated by suboptimal queries. Please see Abhishek’s primer from last week on how to write queries that aren’t terrible.

Charity Majors
April 1, 2014

Production vs Development Apps

prodapps

It’s a very common use-case to have apps built on Parse that are both still in development and already in production. To help you make a clear distinction between Development Apps and Production Apps, we’re adding a switch in your app’s settings that lets you mark an app as a Production App. This will not only allow you to visually differentiate between your production apps and those still in development mode, but will in the future allow us to add security features that prevent you from accidentally editing content that is in production, or sending test push notifications to actual users (“Whoops!”).

We’re pretty excited about this, and hope you’ll find it useful!

Christophe Tauziet
March 31, 2014

Writing Efficient Parse Queries

Any time you store objects on Parse, they get stored on a database. Usually, a database stores a bag of data on disk, and when you issue Parse queries it looks up the data and returns the matching objects. Since it doesn’t make sense for the database to look at all the data present in a particular Parse class for every query, the database uses something called an index. An index is a sorted list of items matching a given criteria. Indexes help because they allow the database to do an efficient search and return matching results without looking at all of the data. For the more algorithmically inclined, this is an O(log(n)) vs O(n) tradeoff. Indexes are typically smaller in size and fit in memory, resulting in faster lookups.

There are a couple of operations – $ne (Not Equal To) and $nin (Not In) which cannot be supported by indexes. When you run a query with a “Not Equal To” clause, the database has to look at all objects in the particular Parse class. Consider an example: we have a column “Color” in our Parse Class. The index helps us know which objects have “Color” blue, green and so on. When running the query:

 { "Color" : { "$ne" : "red" } }

The index is useless, as it can only tell which objects have a particular color. Instead, the database has to look at all the objects. In database language this is called a “Full Table Scan” and it causes your queries to get slower and slower as your database grows in size.

Luckily, most of the time a $ne query can be rewritten as a $in query. Instead of querying for the absence of values, you ask for values which match the rest of the column values. Doing this allows the database to use an index and your queries will be faster.

For example if the “User” object has a column called “State” which has values “SignedUp”, “Verified”, and “Invited”, the slow way to find all users who have used the app at least once would be to run the query:

 { "State" : { "$ne" : "Invited" } }

It would be faster to use the $in version of the query:

 { "State" : { "$in" : [ "SignedUp", "Verified" ] }

In a similar vein a $nin query is bad, too, as it also can’t use an index. You should always try to use the complimentary $in query. Building on the above example, if the “State” column had one more value, “Blocked”, to represent blocked users, a slow query to find active users would be:

 { "State" : { "$nin" : [ "Blocked", "Shadow" ] } }

Using a complimentary $in query will be always be faster:

 { "State" : { "$in" : [ "Active", "Verified" ] } }

We’re working hard here at Parse so that you don’t have to worry about indexes and databases, and knowing some of the tradeoffs in using the $ne, $nin operators goes a long way in making your app faster.

Abhishek
March 27, 2014

A New Way to Report Bugs

If the way to a developer’s heart is great documentation, the way to stay there is through great support. Here at Parse we’ve always been happy to help through our community forums as well as through face to face conversations at one of our many developer meetups. As our community has grown, it has become increasingly important to provide a way to surface sensitive issues in a timely manner. Today, I’d like to introduce a new addition to our Help portal, the Report a Bug button:

reportbug
This will allow you to create a new bug report which can be associated with one of your Parse apps. You will also be able to tag your bug report with one of the top level product categories. This will help us triage bug reports as well as help other Parse developers discover related issues.

Let’s take for example a simple messaging app that allows people to send photos to their friends. These are stored in an Attachments class and I have set up an array of pointers through a Pointer field, attachments, in my Message class. Imagine for a second that the query I am using is suddenly not working and I’ve decided to submit a bug report. This sounds like a great fit for the iOS/OS X SDK > Queries category.

bugtoolreportflow
The above is a great example of a simple but concise bug report. It describes the unexpected behavior and provides all the code needed for our engineers to reproduce the issue. Once submitted, the report will be assigned to one of our engineers, and you will receive notifications on status updates for this specific report, such as “Fixed” or “Needs More Info”.

bugtoolstatus
You can also search for existing bug reports and known issues through the Bug Tool. You may subscribe to existing bug reports should you find that you’re running into a similar issue.

subscribe
I’m really excited about the launch of this new support channel, which will allow us to provide faster turnaround times for your bug reports. As always, feel free to leave any feedback on our community forums!

Héctor Ramos
March 25, 2014

MongoDB 2.6 Production Release Highlights

The Parse platform relies heavily on MongoDB. We use MongoDB for a variety of workloads, such as routing and storing application data, monitoring, real-time API request performance analysis, and billing and backend platform analytics. We love the flexibility and richness of the feature set that MongoDB provides, as well as the rock-solid high availability and primary elections.

So we are incredibly excited about the upcoming MongoDB 2.6.0 production release. You can read the full release notes here, but I’d like to highlight a few of the changes and features we are most excited about.

First and foremost, the index build enhancements. You will now be able to build background indexes on secondaries! For those who are unfamiliar with the way mongo performs indexing, the default behavior is to build indexes in the foreground on both primaries and secondaries. Foreground indexing means the index op will grab the global lock and no other database operations will be able to execute until the index has finished building (this includes killing the index build). Obviously this is not a reasonable option for those of us who ensure indexes are built in the normal request path. You can instruct the primary to build indexes in the background, which lets the indexing operation yield to other ops, but until now there has been no similar functionality on the secondaries. Parse makes extensive use of read queries to secondaries for things like our push services, and we have had to implement a complicated set of health checks to verify that secondaries are not locked and indexing while we try to read from them. Background indexes on secondaries will make this process much simpler and more robust.

Another terrific indexing improvement is the ability to resume interrupted index builds.

We are also very excited about a number of query planner improvements. The query planner has been substantially rewritten in 2.6.0, and we are eager to take it for a test drive. Mongo has also now implemented index intersection, which allows the query planner to use more than one index when planning a query.

The explain() function has been beefed up and they have added a whole suite of methods for introspecting your query plan cache. In the past we have often been somewhat frustrated trying to infer which query plan is being used and why there are execution differences between replica set members so it is great to have these decisions exposed directly.

Another interesting change is that PowerOf2Sizes is now the default storage allocation strategy. I imagine this is somewhat controversial, but I think it’s the right call. PowerOf2 uses more disk space, but is far more resistant to fragmentation.  An ancillary benefit is that padding factors are no longer relevant. One issue we have had at Parse (that no one else in the world seems to have, to be fair) is that we cannot do initial syncs or repairDatabase() because it resets all the padding factors to 1.0. This causes all updates or writes to move bits around disk for weeks to come as the padding factors are relearned, which in turn hoses performance. The inability to do initial sync or repair means we have had no way of reclaiming space from the database.

The hard-coded maxConns limit is also being lifted. Previously your connection limit was set to 70% of ulimit or 20k connections, whichever is lower, but the hard cap is now gone.  This totally makes sense and I am glad it has been lifted. However you should still be wary of piling on tens of thousands of connections, because each connection uses 1mb of memory and you do not want to starve your working set of RAM.

Here’s another thing I missed the first dozen or so times I read through the release notes: QUERY RUNTIME LIMITS! MongoDB now lets you tag a cursor or a command with the maxTimeMS() method to limit the length of time a query is allowed to run. This is a thrilling change. Parse (and basically everyone else who runs mongo at scale) has a cron job that runs every minute and kills certain types of queries that have been running past their useful lifespan (e.g. their querying connection has vanished) or are grabbing a certain type of lock and not yielding. If maxTimeMS() works as advertised, the days of the kill script may be gloriously numbered.

Ok, so those are the delicious goodie bits. Lastly let’s take a look at a painful but necessary change that I am afraid is going to take a lot of people by surprise: stricter enforcement of index key length. In all previous versions of mongo, it would allow you to insert a key with an indexed value larger than 1024 bytes, and it would simply warn you that the document would not be indexed. In 2.6 it will start rejecting those writes or updates by default. This is unquestionably the correct behavior, but will probably be very disruptive for a lot of mongo users when previously accepted writes start to break. They have added a flag to optionally preserve the old behavior, but all mongo users should be thinking about how to move their data sets to a place where this restriction is acceptable. The right answer here is probably some sort of prefix index or hashed index, depending on the individual workload.

There is a lot of exceptionally rich feature development and operational enhancements in this 2.6 release. We have been smoke testing the secondaries in production for some time now and we’re very much looking forward to upgrading our fleet. Be sure to check out the rest of the 2.6 release notes for more great stuff!

Charity Majors
March 25, 2014

Parse Developer Day Video Series: A New Kind of Checkout (Stripe)

This is a guest post from Christian Anderson, Software Engineer at Stripe.

Users abandon purchases for many reasons: poor design of the checkout form, the hassle of having to punch in their card details, or even just the fact that they’re confronted with an unfamiliar checkout experience. With Stripe Checkout, Stripe tackles these problems so you can focus on your core business.

This video from Parse Developer Day shows viewers how to use Checkout and Parse’s Stripe integration to craft a great purchasing experience for your users, including a demo of the powerful Checkout feature.

Courtney Witmer
February 12, 2014

Static File Requests, All by Themselves

It’s always been easy to store a static file on Parse, whether through Parse Files or static assets on Parse Hosting. What we haven’t made easy, though, is knowing what percentage of your traffic is from your app requesting a static file instead of making an intelligent request to our servers.

We’ve traditionally lumped requests for those assets into your overall request usage numbers, visible on your dashboard and used for billing:

File Requests as part of Dashboard Requests

… but only Parse Hosting static file requests were reflected in our analytics graphs.

We’re happy to say that, as of last week, we’ve cleaned the situation up a bit. File Requests are still reflected in your dashboard usage numbers (and used for billing: “Requests” reflect the sum of API Requests and File Requests), but the two are now separated out in analytics graphs.

The default “API Requests” option will only reflect requests that hit our servers (e.g. an object save, user login, etc) while the new “File Requests” option will reflect Parse File and Parse Hosting file requests:

File Requests in Analytics Graphs

We’ve been working busily on giving you clearer, more reliable insight into usage of your Parse apps. Let us know what you think, and keep an eye out for more goodies coming your way!

Christine Yen
February 6, 2014

Parse Developer Day Video Series: Monetize Your Windows Apps

The following is a guest post from Sanjeev Dwivedi, Technical Evangelist at Microsoft.

Windows 8 and Windows Phone 8 offer a fast growing platform for developers to build apps and start monetizing. Apps you build can be made available to all types of devices, from phones to tablets and Windows 8 PCs, giving you a wide reach of users, from casual to enterprise environment. The opportunity is even larger if you publish your apps to the more than 190 markets where Windows and Windows Phone Stores are available. The Stores also support the largest options of payment instruments, to help more consumers purchase your apps or in-app purchase: we support credit card, debit card, PayPal, mobile operator billing, Alipay (China) and others depending on the market. No other store enables you to publish to all three screens, and is available in so many different markets, with as many payment instruments.

On the technical side, Windows and Windows Phone offer developers a choice on whatever technology they know and are familiar with to develop apps. On both Windows 8 and Windows Phone, you can use C++ or C# to develop your applications, and Windows also offers HTML5/JavaScript – each of these languages have full access to all of the API. So, you can develop apps using technologies that you are familiar with and love.

If you want to learn more about the Windows and Windows Phone opportunity, tune in to the video, below.

Courtney Witmer
February 5, 2014