Authors: George Witteman and Rohan Ramchand

In our previous post, we talked about storing money: why it’s hard, where we got it wrong, and what we needed to do about it. We came up with a set of rules we needed to follow:

1. Money looks arbitrarily precise, but it isn’t always. Until you actually need to move money in the real world (e.g. by charging a user), you can represent money internally however you want.

2. However, once you do need to move the money around, you have to round it to the currency's minor unit.

3. Thus, in order to do anything with money, you can’t just know the amount: you also need to know the scale of the currency.

4. So, we can represent money as an object with two fields: an amount and a currency, the latter of which determines the scale.

Here’s what we built to solve the problem.

Storing money at scale

The common wisdom that you should never, ever use floating-point numbers is basically true. IEEE 754 floating points, which most languages use to power their built-in float types, are imprecise, because they have to convert everything to base 2.

But there are other ways to store numbers, and a common approach is to use integers or arrays of integers. Both are a great representation of numbers, because (a) they can be arbitrarily precise (as opposed to floating points, where you basically just have 32 or 64 bits’ worth of precision), and (b) you can do math pretty well with them. Most libraries that use one of these representations under the hood refer to themselves as “decimal” libraries.

We use a popular decimal library, big.js, to represent the amount. big.js allows us to represent arbitrarily precise decimals and handles implementing all of the common mathematical operations you’d expect (addition, square roots, etc.). This gets us most of the way there, but big.js doesn’t exactly represent scale in the terms that we want. 

It’s worth pausing here to be precise about what scale is. Numbers can have whatever scale you’d like them to have—for example, 100 can be represented as “100 with scale 0”, “1 with scale -2”, or anything else. This is a broader and more fluid definition of scale than we’ve seen before—money, for example, usually has a fixed scale, defined by the currency—but because we’re just dealing with arbitrary numbers at this layer, we need to support using any scale for any number, rather than simplifying to, e.g. the scientific notation rule of “coefficients between 1 and 10” with a scale to match.

So, we created an abstraction on top of big.js, called Decimal, which allows any combination of amount and scale. That said, most of the time, we derive the scale from the amount as the number of decimal places in the string representation of the amount. So, for example, "1.00" has a scale of 2, despite being reducible to a scale of 0 without loss of precision. However, we hard-fail if we’re asked to store a number in a way that would lose precision; for example, trying to store 1.234 with a scale of 2 throws an error. This means we can safely carry out whatever operations we’d like to perform, without ever risking losing precision.

It then defines a bunch of operations like addition and subtraction, all of which essentially do variations of the same thing:

1. Convert all of the inputs to Decimal instances.

2. Perform the operation without loss of precision on the underlying Big values.

3. Return a Decimal whose amount is the result of the operation, and whose scale is set to the lowest value we can support without losing precision. (How exactly this minimum “safe” scale is determined is up to each operation; for example, addition sets the output scale to the max of the input scales, while multiplication sets it to the sum.)

So, for example, adding 1.234 (scale 3), 5.67 (scale 2), and 8 (scale 1) yields an amount of 14.904 (the sum of the amounts with all precision preserved) with a scale of 3, the smallest scale that preserves precision in the output. Similarly, multiplying 1.234 and 5.67 yields 6.99678, with a scale of 5, once again the smallest safe scale to preserve precision.

The Decimal class is therefore mostly pretty straightforward: it defines a bunch of ways to create a Decimal (from a number with a scale, from a string, from another Decimal, etc.), and it implements a bunch of basic operations (addition, multiplication, etc) that maintain precision for us.

There’s one operation, however, that’s non-trivial to define: division. In particular, as opposed to all other operations, division allows for outputs with infinite scale—for example, 10 divided by 3 is 3 with an infinite number of 3s after the decimal point. So, because computers are finite, we are forced to lose precision and round to a finite scale.

And so, whereas all of the other operations calculate the scale of the output from the scale of the inputs, the only division operator, helpfully named divideByAndRound, requires passing in a scale, and returns a rounded number.

For the most part, we don’t use Decimal directly, since we have a Money class (see below) that handles most of the use cases for precision operations. There are a few cases, though, where we need to maintain precision in our operations for non-money amounts, like tax calculations or foreign exchange; in these cases, we can use Decimal directly and get all of the safety benefits that it provides.

const usdToGbpExchangeRate = Decimal.fromNumber(0.77184832);
const taxRate = Decimal.fromString("0.0875");
const feePct = Decimal.fromScaledInteger(5, 2); // = 0.05 = 5%

Back to Money

Now that we have a nice class to handle all of our operations with scaled amounts, we can now define our Money class as a wrapper around the Decimal class with a pre-determined scale. Easy, right?

Right! We made our lives really easy by separating out the concepts of “amounts with scale” and “amounts with a predefined, semantically-interesting scale.” There’s only a few minor things that we need to add.

First, unlike amounts, where we can perform operations like addition on operands with different scales, it’s really important that we don’t accidentally add two Money objects with different currencies (even if those currencies have the same scale). So, before every operation, we check that all of the operands have the same currency.

Next, whereas within our system we can represent Money objects with arbitrary precision, we don’t actually want our APIs to take in strings for their amount fields; instead, we still want them to take in integers representing the minor unit of the currency. (This is mostly by convention, but also because taking in integers forces our callers to deal with this exact problem on their side as well, rather than just defaulting to doing something imprecise.) So, we add a few helper functions, like toMinorUnitInt, to make converting API inputs and outputs a little easier.

Finally, division on Money objects is more complicated than division on Decimal objects. In the example above of dividing 10 by 3, the output is 3 with an infinite number of 3s after the decimal point, so the Decimal implementation forces you to pass in a scale to round the output to. In theory, we could just expose Decimal.divideByAndRound on Money and call it a day, like we do with the other arithmetic operations.

However, in most cases, you don’t actually want to divide Money. For example, let’s say that you want to divide a USD$10 discount among three items. divideByAndRound with a scale of 2 will helpfully return $3.33 for each item, but then the user would only receive a total discount of $9.99. What we want, then, is a way to divide a Money object between a set of buckets, in such a way that all of the amount is captured. In fact, ideally, we want to be able to do this even if the buckets aren’t equally sized, for example if one person is covering two shares.

This operation is generally called allocation, and works by first performing simple division on the base amount. Then, any amount left over (the remainder) is allocated, one minor unit at a time, until there’s nothing left over. So, for example, in the example above, allocation starts by dividing the $10 total into three equal buckets of $3.33 each. Because there’s a remainder of $0.01, we add one penny to each bucket until we run out; in this case, that just happens once, and we’re left with one bucket of $3.34, and two buckets of $3.33.

const total = Money.fromDecimalNumber(1400, "JPY");
const fixedFee = Money.fromDecimalNumber(0.05, "USD");
const feePct = Decimal.fromNumber(0.05);
const jpyToUsdRate = Decimal.fromNumber(0.00666098420);
const usdTotal = total.convert("USD", jpyToUsdRate);
const fee = usdTotal.times(feePct).plus(fixedFee).round();

Migrating

It was initially very easy to migrate to the Money class. Any time we passed around an amount representing money, we converted it from a number to a Money. This got us decently far, but only so far as the edges of our codebase.

First, we had to update all of our third-party API calls. For most APIs, this was extremely straightforward, since they mostly all expected integers in the minor unit. However, while doing the migration, we realized (with mild concern) that one of the APIs we used was handling zero-decimal currencies the same way we were—that is, they also pretended all money amounts had scale 2. We reached out to them for support, and ended up helping them upgrade their systems the same way we were doing.

The next issue was our database. Every amount representing money in our database was stored as a scale-2 integer, and converting all of these to the right scale would have taken much more time than we had. So, instead, we decided to keep things as is for the time being, added a from/toNeonInteger helper that fixed the scale to 2, and moved on.

Finally, we’re working on migrating our public APIs, all of which also treat money amounts as scale-2 integers. Here again, we can call from/toNeonInteger as appropriate in the short term, but should be able to derive the scale from the currency correctly after the migration is complete.

And so

This may seem confusing—why go through all of this if we’re not actually changing any of our data?—but the benefit of the new library is that we now assert that money amounts are scaled correctly. Because we represent everything as scale-2 integers, and because for now we only handle scale-0 and scale-2 currencies, this is doable without loss of precision—we can just assert that, for example, all JPY amounts are rounded to the nearest 100.

Then, once we want to handle currencies with a scale greater than 2, the migration will be much more straightforward. We’ll need to version our APIs, but we’ll be able to update our databases in one migration. And, whereas before, we had to manually inspect all of our logs to make sure customers were calling the API with the right amount of padding for scale-0 currencies, we now return a 400.

We’re working on cleaning up some edge cases in this library (for example, did you know that New Taiwan dollars technically has a minor unit of 2, but in practice it's almost never used, and some banks won't allow you to charge decimal amounts?), and hope to open-source this soon. In the meantime, though, if this is the kind of stuff you find interesting, let us know—we’re always hiring engineers to tackle problems like this: the ones that don’t, you know, scale.