2024-11-11 16:00:00

I’ve been working professionally for the better part of a decade on web apps and, in that time, I’ve had to learn how to use a lot of different systems and tools. During that education, I found that the official documentation typically proved to be the most helpful.

Except…Postgres. It’s not because the official docs aren’t stellar (they are!)–they’re just massive. For the current version (17 at the time of writing), if printed as a standard PDF on US letter-sized paper, it’s 3,200 pages long.¹ It’s not something any junior engineer can just sit down and read start to finish.

So I want to try to catalog the bits that I wish someone had just told me before working with a Postgres database. Hopefully, this makes things easier for the next person going on a journey similar to mine.

Note that many of these things may also apply to other SQL database management systems (DBMSs) or other databases more generally, but I’m not as familiar with others so I’m not sure what does and does not apply.

Normalize your data unless you have a good reason not to

Database normalization is the process of removing duplicate or redundant data from a database schema. For example, say you have a site where users can upload documents and users can subscribe to email notifications when folks view their documents. If you have a documents table, you shouldn’t have a user_email column on said table: When a user wants to change their email, you shouldn’t have to update the hundreds of rows for all the documents they’ve uploaded. Instead, you can have each row in documents represent a row in another table (e.g. users) with a foreign key (e.g. user_id).

If you search for “database normalization” online, you’ll find a bunch of results about “1st normal form” and the like. You definitely don’t need to know what each of those “normal forms” are, but the general process is good to be familiar with as it can lead you to a more maintainable database schema.

There are occasions when having redundant data (that is, denormalizing your schema) can make sense: this is typically to make reading certain data faster so it doesn’t have to be recomputed every time it’s requested. E.g. you have an application that manages employee shifts at a bakery. A user might want to see how many hours they’ve worked so far this year. To calculate that, you would need to do something like determine the duration of each of their shifts and then sum all of them together. It might make sense to instead calculate this amount on a regular interval or whenever the number of hours worked changes. This data can be denormalized within the Postgres database or outside of it (e.g. in a caching layer like Redis). Note that there is almost always a cost to denormalized data, whether that’s possible data inconsistency or increased write complexity.

Follow all the advice from the folks that make Postgres

There’s a big list aptly titled “Don’t do this” on the official Postgres wiki. You may not understand all of the things listed. That’s fine! If you don’t understand, then you probably won’t make the mistake. Some suggestions worth highlighting:

1. Just use the text type for all text storage
2. Just use the timestampz/time with time zone type for all timestamp storage
3. Name your tables in snake_case

Note some general SQL eccentricities

Saving your pinkies: you don’t have to write SQL in all caps

In most documentation and tutorials, you’ll see SQL written like so:

SELECT * FROM my_table WHERE x = 1 AND y > 2 LIMIT 10;

The case for SQL keywords does not matter. That means the snippet above is the same as this:

select * from my_table where x = 1 and y > 2 limit 10;

or this:

SELECT * from my_table WHERE x = 1 and y > 2 LIMIT 10;

As far as I know, this is not specific to Postgres. Your pinkies will thank you.

NULL is weird

You’re probably familiar with null or nil values from other programming languages. SQL’s NULL is not like that. NULL is really more accurately an “unknown”. E.g. NULL = NULL returns NULL (because it’s unknown if one unknown equals another unknown!) This is true for almost any operator, not just =(we’ll go through some of the exceptions in a moment): if one side of the comparison is NULL, the result will be NULL.

There are a few operators that you can use to compare against NULL which don’t result in NULL:

WHERE clauses only match if the condition evaluates to true. This means that a query like SELECT * FROM users WHERE title != 'manager' won’t return rows where title is NULL because NULL != 'manager' is NULL.

Another useful function when attempting to wrangle NULL is COALESCE: COALESCE will take any number of arguments and return the first one that is not NULL:

COALESCE(NULL, 5, 10) = 5
COALESCE(2, NULL, 9) = 2
COALESCE(NULL, NULL) IS NULL

You can make psql more useful

Fix your unreadable output

Have you ever done a query on a table with many columns or long values in those columns and the output is basically unreadable? It’s probably because you don’t have the pager enabled. A terminal pager allows you to view a file (or table, in the case of psql) of text by scrolling a viewport around some larger canvas. Without a pager, it just dumps the text in your terminal, wrapping to a new line wherever it runs out of space.

less is a decent pager available on any Unix-like system. You can set it as your pager by setting the environment variable in your ~/.bashrc/~/.zshrc/etc.:

# Use the `-S` option so it truncates long lines for scrolling instead of wrapping them
export PAGER='less -S'

Sometimes, even viewing things as a properly formatted table isn’t particularly useful, especially with tables with many columns. For this, you can switch to “expanded” mode with \pset expanded (or the shortcut \x) in your psql session. If you want to have this as the default, you can add a file called .psqlrc to your home directory (i.e. ~/.psqlrc) and add \x to it. Any time you start up a psql session, it’ll run all the commands in that file first.

Clarify ambiguous nulls

It’s very important to know when a value is NULL in an output, but the default settings don’t make that particularly clear. You can give psql a string to output when it’s referring to NULL. I have it set to [NULL] by running

\pset null '[NULL]'

Any Unicode string is fine! It’s a bit past spooky season, but you can be like my friend Steven Harman and set it to “👻”.

Once again, if you want to have this as the default, you can add a file called .psqlrc to your home directory (i.e. ~/.psqlrc) and add \pset null '[NULL]' to it. Any time you start up a psql session, it’ll run all the commands in that file first.

Use auto-completion

psql, like many interactive consoles, will allow for auto-completion. It helps that SQL is a fairly rigid and fairly structured language. You can just start typing most keywords or table names and hit Tab and let psql fill out the rest:

-- start typing "SEL"
SEL
-- ^ hit `Tab`
SELECT

Lean on backslash shortcuts

There are a whole bunch of useful shortcut commands in psql for looking up stuff, command line editing, and more.

There are a ton of these and the table above only scratches the surface.

Copy to a CSV

Sometimes, you just want to share the output of a command with someone who wants to put it in Excel or something. In Postgres, it’s actually really easy to copy the output of any query to a CSV on your local machine²:

\copy (select * from some_table) to 'my_file.csv' CSV

If you want it to include an initial line with all the column names, you can add the HEADER option:

\copy (select * from some_table) to 'my_file.csv' CSV HEADER

For way more information (including how to get it to do the reverse: insert rows from data in a CSV!), check out the docs on this command.

Use column shorthands and aliases

When doing a SELECT statement in psql, you can re-title each of the output columns to whatever you want (an “alias”) with the AS keyword:

SELECT vendor, COUNT(*) AS number_of_backpacks FROM backpacks GROUP BY vendor ORDER BY number_of_backpacks DESC; 

This will also rename the column in the output.

What’s more is that GROUP BY and ORDER BY have their own nice shorthand as well: you can reference output columns by the number they appear after SELECT. Therefore, you can write the previous query as

SELECT vendor, COUNT(*) AS number_of_backpacks FROM backpacks GROUP BY 1 ORDER BY 2 DESC; 

While useful, don’t put this in any queries that you ship to production–your future self will thank you!

It’s possible that adding an index will do nothing (particularly if it’s misconfigured)

What is an index?

An index is a data structure intended to help with looking up data–giving Postgres the responsibility of maintaining a “shortcut directory” to a table’s rows by various fields. By far the most common kind is a B-tree index, which is a kind of search tree that work for both exact equality conditions (e.g. WHERE a = 3) as well as range conditions (e.g. WHERE a > 5).

But you can’t tell Postgres to use a specific index. It needs to predict (using statistics it maintains for each table) that it’ll be faster than just reading the table from top to bottom to find the relevant data (known as a “sequential scan” or “seq. scan”–pronounced “seek scan”–for short). You can see how Postgres is planning on executing a query by adding EXPLAIN before your SELECT ... FROM .... This will give you a “query plan”: a plan for how Postgres is going to find the data and its estimate for how much work each task will take. There are many good guides for reading the output of these query plans like this one from thoughtbot or this one from pganalyze. The official docs are also a good reference (albeit a bit overwhelming for a newbie). For analyzing query plans, I’ve found this tool to often be very helpful

An index isn’t much use for a table with barely any rows in it

This is particularly important when doing development on your local database. Chances are, you don’t have millions of rows in your local database. Postgres may find that it’s faster just to do a seq. scan rather than use the index at all if it’s just dealing with 100 rows.

When indexing multiple columns, the order matters

Postgres supports multicolumn indexes which do what you might expect: if you create an index on columns a and b like so:

CREATE INDEX CONCURRENTLY ON tbl (a, b);

then a WHERE clause like

SELECT * FROM tbl WHERE a = 1 AND b = 2;

will be faster than if two separate indexes were created for a and b respectively. This is because in one multicolumn index, Postgres just needs to traverse one B-tree which is able to efficiently combine the constraints of the search query.

This index speeds up queries filtering against just a just as much as an index against a alone.

But what about a query like SELECT * FROM tbl WHERE b = 5;? Is that faster too? Possibly, but not as fast as it could be. It turns out the above index does not obviate the need for an index on b alone. That’s because the B-tree in the index is first keyed on a and secondarily keyed on b. So it will need to traverse all of the a values in the index to find all of the b values in the index. Often if you need to use any combination of columns for you queries, you’ll want to have indexes for both (a, b) as well as b alone. That said, you may be able to rely on indexes on a and b separately, depending on your needs.

If doing prefix matches, use text_pattern_ops

Let’s say you’re storing a hierarchical system of directories in your database using a materialized path approach (where you store a list of all of a row’s ancestors’ ids in each row) and for some part of your application you need to get all descendant directories. So you need to query a table to find all rows where a column matches some common prefix:

-- % is a wildcard: the `WHERE` clause here is asking for `directories` where `path` starts with '/1/2/3/'
SELECT * FROM directories WHERE path LIKE '/1/2/3/%'

To keep things speedy, you add an index to the path column of directories:

CREATE INDEX CONCURRENTLY ON directories (path);

Unfortunately, this may not be used: Most kinds of indexes (including the default B-tree index implicitly created in the CREATE INDEX statement above) depend on the ordering of values to work. To make Postgres able to do the basic character-by-character sorting that you need for this sort of prefix matching or pattern matching in general, you need to give it a different “operator class” when defining the index:

CREATE INDEX CONCURRENTLY ON directories (path text_pattern_ops);

Long-held locks can break your app (even ACCESS SHARE)

What is a lock?

A “lock” or “mutex” (short for “mutual exclusion”) ensures only one client can do something dangerous at a time. This is a concept you’ll see in a lot of places but they’re particularly important in Postgres, as in any database, because updating an individual entity (whether that’s a row, table, view, etc.) must entirely succeed or entirely fail. One way that operations could only partially succeed is if two different clients/processes were attempting to perform them at the same time. As a result, any operation will need to acquire a “lock” on the relevant entity.

How locks work in Postgres

In Postgres, there are several different locking levels for tables which are more or less restrictive. Here’s just a few in order of less to more restrictive:

And here’s how they conflict (X means they are conflicting):

For example consider the following for a single table:

For a full list of all of this information, look at the official documentation. This guide is also a great reference to see what conflicts with what on an operation-by-operation basis (which is typically what you’re thinking about, instead of lock levels).

How this can cause problems

In the previous section, we noted that an if one client is performing an ALTER TABLE statement, that can block a SELECT from running. This can be just as bad as it sounds if the ALTER TABLE statement takes a long time. If you’re updating a core table (e.g. users, one that all requests for your web app may need to reference) all SELECTs reading from that table will just be waiting. Before timing out, of course, causing your app to return 503s.

Common recipes for slow ALTER TABLE statements include

• Adding a column with a non-constant default⁴
  • In my experience this is by far the most common cause of slowness
• Changing a column’s type
• Adding a uniqueness constraint

So, let’s say you’re adding a new column to a heavily used table. You aren’t doing anything silly with your ALTER TABLE statements. Sure, you’re adding a new column, but it doesn’t have a variable default. This can still break your app.

See, that ALTER TABLE statement will be fast…once it acquires the lock. But say that, years ago, you made some internal dashboard that does periodic queries against that table. Over time, that query got slower and slower. What once took milliseconds now takes minutes. Which is normally fine–it’s just a SELECT statement after all. But if your ALTER TABLE statement gets executed while one of those is running, it will have to wait.

That all probably isn’t too surprising. But you might find this a bit surprising: Any subsequent statements querying that table will have to wait, too. That’s because Postgres locks form a queue:

For a great article on this exact scenario happening, see here.

Long-running transactions can be just as bad

If you’re not familiar with transactions, they’re a way of grouping together a series of database statements so they act as all-or-nothing (in fancy lingo, they’re “atomic”). Once you begin a transaction (with BEGIN, of course), you’re hiding away. No other clients can see any changes you’re making. You can finish the transaction (with COMMIT) which then “publishes” them to the rest of the database. Transactions are, in an abstract way, similar to locks: they allow you avoid issues with other clients messing up what you’re trying to do.

A classic example of a task begging for transactions is transferring money from one bank account to another. You would want to decrement the balance of one account and increment the balance of the other. If the database goes down or the originating account balance goes negative midway, you want to cancel the whole operation. Transactions can make that possible.

However, you can easily shoot yourself in the foot when writing transactions if you keep them running too long. That’s because once a transaction acquires a lock, it holds onto it until the transaction commits. For example, let’s say Client 1 opened up psql and wrote the following:

BEGIN;
SELECT * FROM backpacks WHERE id = 2;
UPDATE backpacks SET content_count = 3 WHERE id = 2;
SELECT count(*) FROM backpacks;
-- ...

Oops! Client 1 has stepped away from their desk: Someone just brought home some cupcakes! Even though Client 1 is effectively “done” updating the row with id = 2, they still have the lock.⁵ If another client wanted to delete this row, they would run this:

DELETE FROM backpacks WHERE id = 2;
-- ...
-- ?

But it would just hang. It wouldn’t actually delete anything until Client 1 came back and committed the transaction.

You can imagine how this can lead to all kinds of scenarios where clients are holding onto locks for much longer than they need, preventing others from successfully making queries against or updates to the database.

JSONB is a sharp knife

Postgres has a remarkably powerful feature: you can store queryable and efficiently serialized JSON as a value in a row.⁶ In many ways, it makes Postgres have all the strengths of a document-oriented database (like MongoDB, for example) without having to spin up a new service or coordinate between two different data stores.

However, it has its downsides if used improperly.

JSONB can be slower than normal columns

While JSONB is quite flexible, Postgres doesn’t keep track of JSONB columns’ statistics which can mean that an equivalent query against a single JSONB column can be significantly slower than against a set of “normal” columns. Here’s a great blog post with someone demonstrating it making things 2000x slower!

JSONB is not as self-documenting as a standard table schema

A JSONB column can have basically anything in it–that’s one of the main reasons it’s so powerful! But it also means you have few guarantees about how it’s structured. With a normal table, you can look up the schema and see what a query will return. Will a key be written in camelCase? snake_case? Will states be described with boolean true/false? Or with enums like yes/maybe/no? You have no idea with JSONB as it doesn’t have the same static typing that Postgres data normally has.

JSONB Postgres types are a bit awkward to work with

Let’s say you have a table called backpacks with a JSONB column data where you have a brand field. You want to find the JanSport backpacks because you love the early 90s aesthetic. So you write the query:⁷

-- WARNING: Does not work!
select * from backpacks where data['brand'] = 'JanSport';

and you get the error back

ERROR:  invalid input syntax for type json
LINE 1: select * from backpacks where data['brand'] = 'JanSport';
                                                      ^
DETAIL:  Token "JanSport" is invalid.
CONTEXT:  JSON data, line 1: JanSport

What gives? Postgres is expecting the right-hand side type of the comparison to match the left-hand side type; that is, for it to be a correctly formatted JSON document–therefore, it needs to be a JSON object, array, string, number, boolean, or null. Keep in mind that none of these types have anything to do with Postgres types like boolean or integer. And NULL in SQL works very differently from JSONB’s null, which acts more just like a normal type.⁸ To correctly write this query, you need to enable Postgres to be able to do some coercion.

select * from backpacks where data['brand'] = '"JanSport"';
-- ^ This is really equivalent to the following (because Postgres knows the left-hand side is `jsonb`)
select * from backpacks where data['brand'] = '"JanSport"'::jsonb;
-- Alternatively, you could convert the left-hand side to Postgres `text`:
select * from backpacks where data->>'brand' = 'JanSport';

Note the double quotes inside the single quotes. JanSport on its own isn’t valid JSON.

What’s more is there are a bunch more operators and functions that are specific to JSONB and are hard to remember all at once.

Anyway…

Hope you found this useful. Thank you to Lillie Chilen, Monica Cowan, Steven Harman, and KWu for encouragement and feedback on this post. If you have any corrections, feedback, or comments, you can find me on basically all sites as hibachrach.

2020-10-07 15:00:00

In this article I go over three things that, in my mind, would make JavaScript better. None are new ideas. This post is an expansion of a tweet I had when I saw someone asking about improvements for JS. (though probably are impossible for various reasons). I’m going to be primarily speaking about browsers and the web, though much of this might apply to Node.js (though I’m not as familiar with that area so I can’t speak on it confidently).

Versions for JS

Just as a heads up: I’m not really talking about different ECMAScript versions (e.g. ES6, ES2019, etc.) here–I’m talking about how most programming languages refer to versions.

Right now, there are two versions of JS: Strict mode and “sloppy mode”. Feature detection is done dynamically: if a script makes use of any feature or change in the language that isn’t supported by the environment running it, it will error, either silently or loudly depending on what that feature is.

To get around this, developers do one or both of the following:

Strategy 1: Transpile/polyfill to the lowest common denominator

To get around browsers not supporting features, we use tools like Babel to convert JS making use of newer features to JS supported by all or polyfill them with. However, this has a few problems:

1. Most of the time, the transpilation increases file size and parsing time. Here is a dramatic instance of that (not that this is Babel’s fault…). This is despite the fact that most browsers don’t need this extra code.
2. Transpilation requires a target platform. For now, it seems like the defacto standard is ES5, but this is already changing: websites that can afford to drop older browsers like IE11 are doing so (e.g. GitHub). What is the right target? This might get harder to answer once IE11 disappears and we truly only need to support evergreen (i.e. silently auto-updating) browsers. Different browsers might implement features in different orders. Automated tools like browserslist reduce the impact of this point, but they require upkeep. If a website stops development now but stays online, its JS bundle won’t get any faster despite the herd of browsers moving to support the newer features in the source JS.
3. If one takes shortcuts (via options like “loose mode” for various Babel transpilations) you could actually be introducing bugs by fragmenting the underling semantics of a particular feature (though I admit this problem is not super likely).
4. Transpilation does not get around efforts to dramatically evolve the JS language (especially those which remove old baggage). Syntax that is fundamentally incompatible with old and seldom-used features simply can’t be introduced because we can’t break the web.

Strategy 2: Offer different bundles to different platforms based on proxies

The idea is that you can identify what a browser might need based on the version presented in its “user agent” (UA). There’s a whole article on MDN on why this is a bad idea in general. However, this hasn’t stopped influential companies like Twitter from doing it.

Google Developers instead encourages using the support for <script type="module"> as a discriminating factor. This seems a bit better, but of course this is just one test–Safari is not an evergreen browser and so despite it supporting modules, we can’t rely on this to check for support for “generally new feature” availability in the medium or long-term.

How versioning fits in

As I said at the beginning, there already is a versioning scheme for JS. Strict mode changes the behavior of JS scripts in a backwards incompatible way: if you had a script that worked in “sloppy mode”, it might break in strict mode.

However, it doesn’t look like there are any plans to further extend this approach. When “#SmooshGate” (an incident of browsers accidentally breaking sites relying on old JS extensions by adding incompatible features) happened, versioning was suggested by more than one person. After all, with versioned JS, the issue evaporates. Commenters on Hacker News responded to these suggestions, suggesting that supporting multiple distinct versions introduces significant complexity for developers of JS engines. One person even noted

This has been discussed at length and they have decided not to do it. It’s not a missing feature, it’s by design.

There are other negatives to versioning expounded on in this wonderful article, such as the following quoted here:

• Engines become bloated, because they need to implement the semantics of all versions. The same applies to tools analyzing the language (e.g. style checkers such as JSLint).
• Programmers need to remember how the versions differ.
• Code becomes harder to refactor, because you need to take versions into consideration when you move pieces of code.

I can’t speak much to the work of maintaining engines–this is done by engineers far more skilled than myself. My immediate reaction is that managing different versions might enable stricter handling of various code, leading to simplification, though that’s probably a naive perspective.

On the topic of remembering how versions differ, I would say this is simple in comparison to the inconsistent mess of browser compatibility, JS transpilation configuration, and generally frequent change within the ecosystem (though I will be the first to say that the last point has been fairly exaggerated). In other languages, versions change, and this is considered business as usual.

With regards to added difficulty in refactoring, I would say that this again is probably simpler than other things which we do semi-regularly, such as upgrading major versions of important libraries (e.g. jQuery, webpack), and is likely able to be automated. Additionally, the difficulty is highly dependent on the audacity of those at the reins of JavaScript, who, based on the current environment, seem unlikely to cause unnecessary upset.

Everything is an expression

The main area where I wish this were the standard within JS is with if/else, try/catch, and switch/case statements. This is something that I use very frequently within Ruby

Example: if/else

const a =
  if (cond) {
    b
  } else {
    c
  };

Example: try/catch

const a =
  try {
    somethingThatMightFail()
  } else {
    fallbackValue
  };

Example: switch/case

const a =
  switch (b) {
    case 'apple' {
      'fruit'
    }
    case 'broccoli' {
      'vegetable'
    }
    default {
      'unknown'
    }
  };

Its possible this would need to use different keywords to replace case and default for the sake of JS interpreters and maximizing backwards compatibility because cases function as labels.

Current proposals

To achieve this, do expressions were proposed 2 years ago, which satisfy the requirements with slightly more verbose syntax. E.g. for the if/else, you’d write

const a = do {
  if (cond) {
    b
  } else {
    c
  }
};

However, the proposal is still at stage 1 of the 4-stage TC39 (effective JS steering committee) process, though it’s still being discussed. Some have asked “why do we need the do?” and make the first syntax (without the explicit do) part of the language, though this can’t be done without interfering with existing uncommon language features (another example of not being able to change syntax due to “version constraint”)

Improved caching

I would explain this, but there are actually people already solving this problem, and they’ve put together this article explaining how caching can be improved around the web.

However, there are still unresolved issues here: how does this work for different browser targets? It’s fine if all libraries are only using browser-supported features, but we all know that that won’t be consistent for all features across all browsers into the future. A lot of this builds on the issues presented in the section on versioning above: if there are no ways to talk consistently about versioning, then it’s much harder to solve these problems in an automated way.

2020-04-24 15:00:00

“Where should this information live?” is a common question I encounter daily as a software engineer. I (alone or in consultation with others) will make a decision and that information needs to persist…somewhere. The goal of course is to prevent some future engineer (often myself) from encountering some code and asking “why?” or, even worse, “what?”.

I’ve found there’s a sort of hierarchy to the accessibility of various information that I found useful in sharing.

In order of most accessible to least, the hierarchy is as follows:

1. Code (identifiers)
2. In-code comments
3. Commit messages
4. Pull/merge requests & issue trackers

This list may be different for other people and may change in time as tools get better, but for now, this has held true for me.

I’m sure that similar things have also been said before. Shoulders of giants and all that—would love to read any previous writings on similar topics!

So! Let’s get into some of the reasoning…

Code (Identifiers)

Accessibility

By far, the most important place to put information is in the code itself. Why?

• It can be easily searched with tools like grep, ag, ripgrep, or your favorite file searcher
• It (generally) follows a predictable structure and connections between components can be traced (e.g. through the call stack)
• It is very unlikely to be discarded accidentally

Responsibilities

Because of their inherent visibility, identifiers¹ should always be the first place to put information, especially

• What values represent
• What functions do & how they do it
• (Sparingly) when functions are invoked²

Identifiers and Code Simplification

I’m going to intentionally give you no context for the following code:

if (
  userSession['signed']['admin']['update'] ||
  AdminOverrides[userSession['signed']['id']]
) {
  repository.delete();
  return render({
    status: 200,
    body: {
      notice: 'Repository successfully deleted.'
    }
  });
} else {
  return render({
    status: 403,
    body: {
      errors: ['You are not allowed to perform that action.']
    }
  });
}

Okay, pretty straightforward. But consider this version instead:

const userHasUpdateAccess = (
  userSession['signed']['admin']['update'] ||
  AdminOverrides.update[userSession['signed']['id']]
);

if (userHasAdminUpdateAccess) {
  repository.delete();
  return render({
    status: STATUS_CODES.ok,
    body: {
      notice: 'Repository successfully deleted.'
    }
  });
} else {
  return render({
    status: STATUS_CODES.unauthorized,
    body: {
      errors: ['You are not allowed to perform that action.']
    }
  });
}

We’ve made a few updates. First, we’ve pulled the condition out into its own variable. What that’s done is document the intention of what the condition is supposed to mean. This has a few benefits:

• Each line is doing less, cognitively. That means you have to think about less when reading. You’re going to read code way more than you’re going to write it—why make it harder for yourself?³
• Pulling that connection out and labeling it makes it harder to miss when changes happen. At some point, how we determine whether a user has update access will probably change. Additionally, the body of the first branch of the if may get much larger. It’s always easier to keep things tidy one piece at a time (as opposed to going back and cleaning it up later)!

The other two changes are to replace the HTTP status codes (which are essentially magic numbers) with references to a global STATUS_CODES object. While HTTP status codes are fairly well known (e.g. a “404” is meaningful to many that have never done web development), they’re likely not going to be better known than English.

Generally, breaking things down like this makes the answer to “why?” very obvious.

Caveat: Enigmatic language syntax

Unlike libraries which can often be worked around, specific hard-to-parse features of a language’s syntax may be unavoidable. For example, most languages have regular expressions which are infamous for how easy they are to mess up. Consider the following regular expression in JavaScript:

const IS_VALID_EMAIL_REGEX = /^[^@]+@[^@]+$/;

There’s not really much more we can do in terms of adding/changing identifiers to clarify how the regular expression determines which email addresses are valid and which aren’t–you just have to know the specific syntax of (JavaScript) regular expressions. In cases like these, an in-code comment (which we’ll talk about more in the next section) is the next best place to put this information:

// Matches one or more non-`@` characters, folloed by a `@` character, followed
// by one or more non-`@` characters.
const IS_VALID_EMAIL_REGEX = /^[^@]+@[^@]+$/;

In-code comments

Accessibility

One of the points made about code also apply to in-code comments: they are easily searchable using a file searcher. Also, they do not need any further seeking out than code,⁴ unlike something like commit messages.

However, as they are in a non-programming language, they can be a bit less predictable in terms of structure—you’ll have a harder of a time breaking an English sentence into an abstract syntax tree than compared with something like JavaScript!

Additionally, in-code comments are related to their subjects typically by proximity alone. If someone moves that code without taking the comment with it, the comment ceases to be helpful (or, worse, becomes confusing/misleading). We talk about this more in the “Pitfall” section below.

Responsibilities

Comments, while still being very visible, afford a much greater level of flexibility in comparison with code identifiers for documenting behavior, decisions, etc. Comments are where you put

• Explanations of why a section of code is the way that it is
• Explanations of why a section of code isn’t written another way if one may be tempted to rewrite it
• Contextual information for where a code snippet came from if possibly helpful to future readers⁵
• Explanations of enigmatic language syntax
• (Sparingly) warnings and assumptions⁶

Common Pitfall: Poor Comment Location

For example, say we have this method:

// Must be done in reverse order to be done in O(N) time due to
// Kleppner's Law of Stupid Data Structures
function smooshify(newElements) {
  for (let i = newElements.length - 1; i >= 0; i--) {
    this.flarbo.insert(newElements[i])
  }
}

At some later point, it turns out are some null elements that need to be screened out:⁷

// Must be done in reverse order to be done in O(N) time due to
// Kleppner's Law of Stupid Data Structures
function smooshify(newElements) {
  const nonNullNewElements = newElements.withoutNulls();
  for (let i = newElements.length - 1; i >= 0; i--) {
    this.flarbo.insert(nonNullNewElements[i])
  }
}

With the code as it is now, another dev may stumble along one day and ask “Is withoutNulls iterating in reverse order?” While this may be a bit of a trivial example, these things can lead to a lot more headache down the line

The only way to truly avoid mistakes like this is by is by keeping a vigilant eye on comments. Some techniques I’ve found to be helpful:

• Keep comments as closely positioned to the code they’re describing as possible. E.g. if talking about code in the first branch of an if statement, put that comment in the first branch, not above the entire statement:

  // comment about why `floarp` should be used --- BAD
  if (somethingOrOther()) {
    // comment about why `floarp` should be used --- GOOD
    floarp();
  } else {
    blarg();
  }
  

• Make references to specific identifiers in comments as these tend to garner more attention and help establish a more explicit relationship between the comment and the code

Commit Messages

Most code that I’ve interacted with is version-controlled with something like Git, and if not Git, some other version-control software like Mercurial. I’m going to be referring to Git exclusively in this section, but I’m sure that there are similar tools for other version-control software.

Accessibility

Unlike code or in-code comments, commit messages are not immediately visible when browsing a codebase. However, they can be made much more easily accessible with tools like Fugitive (for Vim) or GitLens (for VSCode). However, they can be searched–just run git log and press / and BAM! You can search through all your commit history (thanks to less)

Unfortunately, commit messages can sometimes be squashed or lost during a rebase, so the information placed there may not be preserved forever, though this mostly depends upon the other collaborators working on the repository.

Responsibilities

Much has been written about how to write good commit messages, so I won’t be too comprehensive here. However, I’d say the things I’m talking about here are, in my opinion, the most important things about writing a useful commit message. As much as I like consistency, the grammatical aspects of commit message guidelines are not nearly as helpful as guidelines on the content of the message (as long as it’s readable!).

Commit messages are all about change. A commit itself is an object representing a change to a codebase and commit messages ought to do the following:

• Briefly summarize what changed, from a high level
• Explain why the changes in the commit are necessary
• Explain why other possible strategies were not attempted, if relevant

These are much easier to accomplish when the commits you make are small and focused (within reason).⁸

Tip: using references to other commits

While commit messages allow you to tell a story, any single commit rarely allows you to tell the whole story. If referencing earlier changes, I highly encourage you to make such references by using the commit hash. This will enable those reading those commit messages later to gain the same context you had when writing the message in the first place with a simple git show. A word of warning however: commit hashes will change when you rebase or amend a commit, so if referencing an earlier commit in a commit message, make sure to update that hash if the earlier commit is ever altered.

Pull/merge requests & issue trackers

These are your GitHubs, your Asanas, your Jiras. I’m lumping issue trackers and hosted version control together because some of these services function as both (e.g. GitHub).

Accessibility

While there’s a large amount of variance here, I always have found messages and comments to get lost somehow, even when generally good search functionality is present. Some services are better than others at this, but even in GitHub (which automatically doubly-link issues to pull requests), comments will be collapsed or hidden in large pull requests, making them a pain to excavate.

Responsibilities

These services are all about facilitating conversation and communication. As such, their responsibilities are focused on those topics:⁹

• Documenting why certain changes weren’t done and/or aren’t worth doing
  • However, these should instead be put in the relevant commit messages if they are related to certain changes that did make it into the codebase
• Containing or summarizing deliberation about proposed changes, even when that deliberation happens in real life or on something like Slack

Conclusion

Hope you found this helpful in answering the question “where do I put this information?”. If you have any thoughts, I’m on Twitter.

2018-04-01 15:00:00

Core philosophy

Make your tools as ergonomic as possible

Ergonomics often comes up when talking with HR during onboarding: it’s the reason why you should use comfortable, supportive chairs, keyboards, mice, and such. “Battle scars” from work shouldn’t be encouraged. If your eyes, wrists, or butt cheeks are hurting, try to address that problem. You’re doing labor. Labor that most likely isn’t as tolling, as say, construction, but labor nonetheless. Take care of your body. If you try to fight it, it’s likely to win. Even if you’re not feeling active pain now, you might start experiencing symptoms after years of damage. “It works for me” shouldn’t be the end of it. Get an ergonomic chair, keyboard, and mouse to reduce strain. Here¹² are some good guidelines on reducing eyestrain (I can personally vouch for tools like f.lux!).

What does this mean for software? If you don’t like the way you’re interacting with your tools (physical or virtual), you’re not going to have as much fun using them.

Sometimes that can be a good thing: if a tool shouldn’t be used, then you can design your system with that idea in mind. As we are all designers in some way, this should be a familiar idea—e.g. if you add a feature that forces a user to do 10 clicks instead of one, they’re less likely to use that feature (c.f. trying to close your Amazon account).

Notice that I’m not talking about speed. The amount of time that you save with one configuration or another probably isn’t going to save you the amount of time that you put into setting it up (unless something was really wrong before-hand). But reducing UX friction can often make you more productive, reduce extraneous repetitive motion, and eliminate unnecessary mental work to get from point A to point B. Some things that help towards this end include:

• Reducing mouse usage³
• Reducing chorded shortcuts (e.g. minimizing stuff like shift+⌘+option+t)
• Reducing the steps to switch between tools and contexts

Make your tools accessible to an outsider without a large amount of reconfiguration

Everything that you do to configure your machine is most likely increasing the barrier to entry somehow. That’s okay. It’s your machine—it should be most comfortable for you. However, that doesn’t mean there won’t be a time where someone else might have to use your keyboard or read your monitor. It should be easy enough for that visitor to understand what’s going on or do some simple tasks without you restarting your damn machine. Note that that doesn’t mean disabling all of your config when someone walks over⁴.

Don’t overdo it

Your tools or configuration won’t make you a significantly better developer. It might even make you a more annoying developer if you won’t stop talking about such topics. And it always takes time to customize these things—the increased efficiency won’t make up for that. Again, it’s about getting you to enjoy using the tools while reducing strain, whether that strain is physical or mental.

tmux

I use tmux inside of iTerm2⁵. Tmux is a “terminal multiplexer” which is an incredibly geeky way of saying it allows you to have windows/splits inside of a OS level window. Here are my main motivations for using it.

Attach and detach

When you start up tmux you’re starting up a server, on which you create and attach “sessions” (basically just workspaces) where you can create windows (really closer to tabs) which can have splits. These sessions can be attached to and detached from at will. And that action is a completely distinct one from opening or closing iTerm2, meaning if iTerm2 has to close for one reason or another, my windows and splits set up exactly as when I left them.

This is great on a local machine but invaluable on remote sessions: when I was doing research on machines that could only be SSH’d into, a random drop in the Internet connection just meant I had to reattach to the session that was running on the remote server. Note that the server will still die if your computer has to restart.

I also find that this navigation is much easier than fiddling with actual windows in a GUI. Which tend to be pretty poorly organized for search and retrieval. Even in macOS, if I have a window that is minimized, I have to either use my mouse to inspect my collection of hidden windows or cycle through them using control+f4⁶.

Prefixes over chords

This is going to be a bit of a recurring theme, but tmux allows you to go from split to split, window to window, or session to session without pressing many keys simultaneously. Instead, you can define a common prefix (I have control+space) and then press the tmux-specific shortcut of your choosing.

How to get started

If you’re on a Mac, you can use Homebrew:

brew install tmux

Here’s a tutorial to get you started.

Vim/Neovim

I could talk probably all day about Vim and why I love it. I’ll try to summarize here to save your time and/or sanity. I will note that Vim has a fairly steep learning curve but I find is worth it in the end. See here for a great blogpost about Vim and usability.

Ubiquity

Vi or Vim is on basically every *nix machine that you might come across. If you know how to use it in one place, you can use it anywhere.

Keyboard-first design

Every interaction with Vim is intended to be done with the keyboard. There are some things that can be done with the mouse (like scrolling or cursor movement), but it’s primarily designed to be manipulated via the keyboard. That makes it generally quite fast, eliminating the time to move your hand to the mouse and back.

Composability

This is probably the biggest and most Vim-specific point here. When you manipulate text in Vim, you do so in a sort of “language”, complete with verbs, nouns, adjectives, adverbs, etc.

Let’s say I’m editing a Ruby script and I’ve got this double negative here. We want to make this a bit more readable, replacing the unless with an if and the empty? with a present?.

Here are all the key presses I needed to do to perform the above:

ciwifjjwwwct?presentjj:wq

Let’s break that down:

• ciw: This is basically a “sentence”. I’m pressing these keys them in what is called “normal mode”. c means “change”, i means “inner”, and w means “word”. “Change” means delete and then go into “insert mode”, the mode where when I type, those characters appear on screen. I applied that “verb” to what is called a “text object”. Here that “text object” is made up of the adjective “inner” with the noun “word”.
• if: This is just me typing the word “if”.
• jj: This is a command I have set up for “insert mode” that gets me back to “normal mode”. This is escape by default but I find this much easier.
• www: Here I’m pressing that “noun” w three times. Since there’s no “verb” here, that just moves my cursor 1 “word” each time. I could have also typed 3w and achieved the same thing.
• ct?: Here I’m using the same c verb but this time with the “noun” t? which means “to ?”, i.e. all the characters from my current position up to (but not including) the next incidence of the character ? on the current line.
• present: This is just me typing the word “present”.
• jj: Same as before.
• :wq: Here I’m performing what is known as a “command”. : gets me into “command mode” (moving my cursor temporarily to the bottom of the Vim window) where I then type w for “write” (meaning “save”) and q for “quit”.

This is a simple example, but as you can see, you can mix and match all these building blocks to manipulate your files in quite complex ways. Many Vim users also create plugins that add core “parts of speech” to the editor. E.g. vim-surround enables the sequence cs"' which allows you to “change the surrounding characters” (cs) from " to ' as well as the sequence ysiw] “add surrounding characters” (ys) to the “inner word” (iw), specifically brackets (]).

Integration with the terminal

(Partly thanks to tmux) I can run Vim in the same program that has my foreman procs, my git interface, my logs, and pretty much everything else. Switching between all of those things is super easy thanks to a plugin I use called vim-tmux-navigator. Basically I can use

• control+h (left)
• control+j (down)
• control+k (up)
• control+l (right)

to navigate between tmux and Vim panes indiscriminately.

When I need to run a test, I have a plugin called vim-test that allows me to run a test in a neighboring tmux pane and then interact with that terminal session afterwards.

There are many other niceties here as well (e.g. sending the contents of a SQL query edited in a Vim session to an interactive psql session) that I may go into in a later post.

So much more

Again, I can talk about Vim all day (and probably will). There are many things not covered here that I will return to later (possibly including a breakdown of my .vimrc).

How to get started

Like I said before, Vim is famous for having a bit of a steep learning curve. I’d recommend trying it out in your leisure time because it will slow you down at first.

Once again, the download for the newest version (probably more up to date than what you have on your machine already) is on Homebrew as well. However, I’d recommend installing the relatively recent fork neovim instead (it has a few nicities including better built-in tutorial:

brew install neovim

Launch it from the shell (using the name nvim, not neovim). Then I’d follow the built-in tutorial by typing :Tutor. The first online resource given at the bottom of the tutorial, “Learn Vim Progressively” is also great—it’s how I first started.

FZF

FZF is a fuzzy-finder for the shell that you can use in a bunch of different ways, like…

Finding files (while also obeying your .gitignore)⁷

Finding git commits

Finding old commands

I couldn’t manage to record this one without any potentially sensitive information, but it works as you might expect.

At its core, FZF is just a way of filtering data in a human-friendly way. You can pipe anything in and get the same fuzzy-find interface (e.g. cat file.txt | fzf).

How to get started

Once again, if you can, just use Homebrew.

brew install fzf

See the project’s installation instructions for details, including howt to integrate FZF with tmux and vim. These integrations work incredibly well so I’d highly recommend setting that up.

That’s it for now! More posts on this stuff to come, I’m sure.