2024-11-11 16:00:00
I’ve been working professionally for the better part of a decade on web apps and, in that time, I’ve had to learn how to use a lot of different systems and tools. During that education, I found that the official documentation typically proved to be the most helpful.
Except…Postgres. It’s not because the official docs aren’t stellar (they are!)–they’re just massive. For the current version (17 at the time of writing), if printed as a standard PDF on US letter-sized paper, it’s 3,200 pages long.1 It’s not something any junior engineer can just sit down and read start to finish.
So I want to try to catalog the bits that I wish someone had just told me before working with a Postgres database. Hopefully, this makes things easier for the next person going on a journey similar to mine.
Note that many of these things may also apply to other SQL database management systems (DBMSs) or other databases more generally, but I’m not as familiar with others so I’m not sure what does and does not apply.
Database normalization is the process of removing duplicate or redundant data from a database schema.
For example, say you have a site where users can upload documents and users can subscribe to email notifications when folks view their documents.
If you have a documents
table, you shouldn’t have a user_email
column on said table:
When a user wants to change their email, you shouldn’t have to update the hundreds of rows for all the documents they’ve uploaded.
Instead, you can have each row in documents
represent a row in another table (e.g. users
) with a foreign key (e.g. user_id
).
If you search for “database normalization” online, you’ll find a bunch of results about “1st normal form” and the like. You definitely don’t need to know what each of those “normal forms” are, but the general process is good to be familiar with as it can lead you to a more maintainable database schema.
There are occasions when having redundant data (that is, denormalizing your schema) can make sense: this is typically to make reading certain data faster so it doesn’t have to be recomputed every time it’s requested. E.g. you have an application that manages employee shifts at a bakery. A user might want to see how many hours they’ve worked so far this year. To calculate that, you would need to do something like determine the duration of each of their shifts and then sum all of them together. It might make sense to instead calculate this amount on a regular interval or whenever the number of hours worked changes. This data can be denormalized within the Postgres database or outside of it (e.g. in a caching layer like Redis). Note that there is almost always a cost to denormalized data, whether that’s possible data inconsistency or increased write complexity.
There’s a big list aptly titled “Don’t do this” on the official Postgres wiki. You may not understand all of the things listed. That’s fine! If you don’t understand, then you probably won’t make the mistake. Some suggestions worth highlighting:
text
type for all text storagetimestampz
/time with time zone
type for all timestamp storageIn most documentation and tutorials, you’ll see SQL written like so:
SELECT * FROM my_table WHERE x = 1 AND y > 2 LIMIT 10;
The case for SQL keywords does not matter. That means the snippet above is the same as this:
select * from my_table where x = 1 and y > 2 limit 10;
or this:
SELECT * from my_table WHERE x = 1 and y > 2 LIMIT 10;
As far as I know, this is not specific to Postgres. Your pinkies will thank you.
NULL
is weirdYou’re probably familiar with null
or nil
values from other programming languages.
SQL’s NULL
is not like that.
NULL
is really more accurately an “unknown”.
E.g. NULL = NULL
returns NULL
(because it’s unknown if one unknown equals another unknown!)
This is true for almost any operator, not just =
(we’ll go through some of the exceptions in a moment):
if one side of the comparison is NULL
, the result will be NULL
.
There are a few operators that you can use to compare against NULL
which don’t result in NULL
:
Operation | Description |
---|---|
x IS NULL |
returns true if x evaluates to NULL , false otherwise |
x IS NOT NULL |
returns true if x does not evaluate to NULL , false otherwise |
x IS NOT DISTINCT FROM y |
the same as x = y but NULL is treated as a normal value |
x IS DISTINCT FROM y |
the same as x != y /x <> y but NULL is treated as a normal value |
WHERE
clauses only match if the condition evaluates to true
.
This means that a query like SELECT * FROM users WHERE title != 'manager'
won’t return rows where title
is NULL
because NULL != 'manager'
is NULL
.
Another useful function when attempting to wrangle NULL
is COALESCE
:
COALESCE
will take any number of arguments and return the first one that is not NULL
:
COALESCE(NULL, 5, 10) = 5
COALESCE(2, NULL, 9) = 2
COALESCE(NULL, NULL) IS NULL
psql
more usefulHave you ever done a query on a table with many columns or long values in those columns and the output is basically unreadable?
It’s probably because you don’t have the pager enabled.
A terminal pager allows you to view a file (or table, in the case of psql
) of text by scrolling a viewport around some larger canvas.
Without a pager, it just dumps the text in your terminal, wrapping to a new line wherever it runs out of space.
less
is a decent pager available on any Unix-like system.
You can set it as your pager by setting the environment variable in your ~/.bashrc
/~/.zshrc
/etc.:
# Use the `-S` option so it truncates long lines for scrolling instead of wrapping them
export PAGER='less -S'
Sometimes, even viewing things as a properly formatted table isn’t particularly useful, especially with tables with many columns.
For this, you can switch to “expanded” mode with \pset expanded
(or the shortcut \x
) in your psql
session.
If you want to have this as the default, you can add a file called .psqlrc
to your home directory (i.e. ~/.psqlrc
) and add \x
to it.
Any time you start up a psql
session, it’ll run all the commands in that file first.
It’s very important to know when a value is NULL
in an output, but the default settings don’t make that particularly clear.
You can give psql
a string to output when it’s referring to NULL
.
I have it set to [NULL]
by running
\pset null '[NULL]'
Any Unicode string is fine! It’s a bit past spooky season, but you can be like my friend Steven Harman and set it to “👻”.
Once again, if you want to have this as the default, you can add a file called .psqlrc
to your home directory (i.e. ~/.psqlrc
) and add \pset null '[NULL]'
to it.
Any time you start up a psql
session, it’ll run all the commands in that file first.
psql
, like many interactive consoles, will allow for auto-completion.
It helps that SQL is a fairly rigid and fairly structured language.
You can just start typing most keywords or table names and hit Tab and let psql
fill out the rest:
-- start typing "SEL"
SEL
-- ^ hit `Tab`
SELECT
There are a whole bunch of useful shortcut commands in psql
for looking up stuff, command line editing, and more.
Command | What it does |
---|---|
\? |
List all of the shortcuts |
\d |
Shows list of relations (tables and sequences) as well as said relation’s owner |
\d+ |
Same as \d but also includes the size and some other metadata |
\d table_name |
Shows the schema of a table (list of columns, including said column’s type, nullability, and default) as well as any indexes or foreign key constraints on said table |
\e |
Opens your default editor (set as the $EDITOR environment variable) to edit your query there |
\h SQL_KEYWORD |
Get syntax and link to docs for SQL_KEYWORD
|
There are a ton of these and the table above only scratches the surface.
Sometimes, you just want to share the output of a command with someone who wants to put it in Excel or something. In Postgres, it’s actually really easy to copy the output of any query to a CSV on your local machine2:
\copy (select * from some_table) to 'my_file.csv' CSV
If you want it to include an initial line with all the column names, you can add the HEADER
option:
\copy (select * from some_table) to 'my_file.csv' CSV HEADER
For way more information (including how to get it to do the reverse: insert rows from data in a CSV!), check out the docs on this command.
When doing a SELECT
statement in psql
, you can re-title each of the output columns to whatever you want (an “alias”) with the AS
keyword:
SELECT vendor, COUNT(*) AS number_of_backpacks FROM backpacks GROUP BY vendor ORDER BY number_of_backpacks DESC;
This will also rename the column in the output.
What’s more is that GROUP BY
and ORDER BY
have their own nice shorthand as well:
you can reference output columns by the number they appear after SELECT
.
Therefore, you can write the previous query as
SELECT vendor, COUNT(*) AS number_of_backpacks FROM backpacks GROUP BY 1 ORDER BY 2 DESC;
While useful, don’t put this in any queries that you ship to production–your future self will thank you!
An index is a data structure intended to help with looking up data–giving Postgres the responsibility of maintaining a “shortcut directory” to a table’s rows by various fields.
By far the most common kind is a B-tree index, which is a kind of search tree that work for both exact equality conditions (e.g. WHERE a = 3
) as well as range conditions (e.g. WHERE a > 5
).
But you can’t tell Postgres to use a specific index.
It needs to predict (using statistics it maintains for each table) that it’ll be faster than just reading the table from top to bottom to find the relevant data
(known as a “sequential scan” or “seq. scan”–pronounced “seek scan”–for short).
You can see how Postgres is planning on executing a query by adding EXPLAIN
before your SELECT ... FROM ...
.
This will give you a “query plan”: a plan for how Postgres is going to find the data and its estimate for how much work each task will take.
There are many good guides for reading the output of these query plans like this one from thoughtbot or this one from pganalyze.
The official docs are also a good reference (albeit a bit overwhelming for a newbie).
For analyzing query plans, I’ve found this tool to often be very helpful
This is particularly important when doing development on your local database. Chances are, you don’t have millions of rows in your local database. Postgres may find that it’s faster just to do a seq. scan rather than use the index at all if it’s just dealing with 100 rows.
Postgres supports multicolumn indexes which do what you might expect:
if you create an index on columns a
and b
like so:
CREATE INDEX CONCURRENTLY ON tbl (a, b);
then a WHERE
clause like
SELECT * FROM tbl WHERE a = 1 AND b = 2;
will be faster than if two separate indexes were created for a
and b
respectively.
This is because in one multicolumn index, Postgres just needs to traverse one B-tree which is able to efficiently combine the constraints of the search query.
This index speeds up queries filtering against just a
just as much as an index against a
alone.
But what about a query like SELECT * FROM tbl WHERE b = 5;
?
Is that faster too?
Possibly, but not as fast as it could be.
It turns out the above index does not obviate the need for an index on b
alone.
That’s because the B-tree in the index is first keyed on a
and secondarily keyed on b
.
So it will need to traverse all of the a
values in the index to find all of the b
values in the index.
Often if you need to use any combination of columns for you queries,
you’ll want to have indexes for both (a, b)
as well as b
alone.
That said, you may be able to rely on indexes on a
and b
separately,
depending on your needs.
text_pattern_ops
Let’s say you’re storing a hierarchical system of directories in your database using a materialized path approach (where you store a list of all of a row’s ancestors’ ids in each row) and for some part of your application you need to get all descendant directories. So you need to query a table to find all rows where a column matches some common prefix:
-- % is a wildcard: the `WHERE` clause here is asking for `directories` where `path` starts with '/1/2/3/'
SELECT * FROM directories WHERE path LIKE '/1/2/3/%'
To keep things speedy, you add an index to the path
column of directories
:
CREATE INDEX CONCURRENTLY ON directories (path);
Unfortunately, this may not be used:
Most kinds of indexes
(including the default B-tree index implicitly created in the CREATE INDEX
statement above)
depend on the ordering of values to work.
To make Postgres able to do the basic character-by-character sorting that you need for this sort of prefix matching or pattern matching in general,
you need to give it a different “operator class” when defining the index:
CREATE INDEX CONCURRENTLY ON directories (path text_pattern_ops);
ACCESS SHARE
)A “lock” or “mutex” (short for “mutual exclusion”) ensures only one client can do something dangerous at a time. This is a concept you’ll see in a lot of places but they’re particularly important in Postgres, as in any database, because updating an individual entity (whether that’s a row, table, view, etc.) must entirely succeed or entirely fail. One way that operations could only partially succeed is if two different clients/processes were attempting to perform them at the same time. As a result, any operation will need to acquire a “lock” on the relevant entity.
In Postgres, there are several different locking levels for tables which are more or less restrictive. Here’s just a few in order of less to more restrictive:
Lock Mode | Example Statements |
---|---|
ACCESS SHARE |
SELECT |
ROW SHARE |
SELECT ... FOR UPDATE |
ROW EXCLUSIVE |
UPDATE , DELETE , INSERT
|
SHARE UPDATE EXCLUSIVE |
CREATE INDEX CONCURRENTLY |
SHARE |
CREATE INDEX (not CONCURRENTLY ) |
ACCESS EXCLUSIVE |
Many forms of ALTER TABLE and ALTER INDEX
|
And here’s how they conflict (X means they are conflicting):
Requested Lock Mode | Existing Lock Mode | |||||||
---|---|---|---|---|---|---|---|---|
ACCESS SHARE |
ROW SHARE |
ROW EXCL. |
SHARE UPDATE EXCL. |
SHARE |
ACCESS EXCL. |
|||
ACCESS SHARE |
X | |||||||
ROW SHARE |
X | |||||||
ROW EXCL. |
X | X | ||||||
SHARE UPDATE EXCL. |
X | X | X | |||||
SHARE |
X | X | X | |||||
ACCESS EXCL. |
X | X | X | X | X | X |
For example consider the following for a single table:
Client 1 is doing… | Client 2 wants to do a … | Can Client 2 start? |
---|---|---|
UPDATE |
SELECT |
✅ Yes |
UPDATE |
CREATE INDEX CONCURRENTLY |
🚫 No, must wait |
SELECT |
CREATE INDEX |
✅ Yes |
SELECT |
ALTER TABLE |
🚫 No, must wait3 |
ALTER TABLE |
SELECT |
🚫 No, must wait3 |
For a full list of all of this information, look at the official documentation. This guide is also a great reference to see what conflicts with what on an operation-by-operation basis (which is typically what you’re thinking about, instead of lock levels).
In the previous section, we noted that an if one client is performing an ALTER TABLE
statement, that can block a SELECT
from running.
This can be just as bad as it sounds if the ALTER TABLE
statement takes a long time.
If you’re updating a core table (e.g. users
, one that all requests for your web app may need to reference) all SELECT
s reading from that table will just be waiting.
Before timing out, of course, causing your app to return 503s.
Common recipes for slow ALTER TABLE
statements include
So, let’s say you’re adding a new column to a heavily used table.
You aren’t doing anything silly with your ALTER TABLE
statements.
Sure, you’re adding a new column, but it doesn’t have a variable default.
This can still break your app.
See, that ALTER TABLE
statement will be fast…once it acquires the lock.
But say that, years ago, you made some internal dashboard that does periodic queries against that table.
Over time, that query got slower and slower.
What once took milliseconds now takes minutes.
Which is normally fine–it’s just a SELECT
statement after all.
But if your ALTER TABLE
statement gets executed while one of those is running, it will have to wait.
That all probably isn’t too surprising. But you might find this a bit surprising: Any subsequent statements querying that table will have to wait, too. That’s because Postgres locks form a queue:
For a great article on this exact scenario happening, see here.
If you’re not familiar with transactions, they’re a way of grouping together a series of database statements so they act as all-or-nothing (in fancy lingo, they’re “atomic”).
Once you begin a transaction (with BEGIN
, of course), you’re hiding away.
No other clients can see any changes you’re making.
You can finish the transaction (with COMMIT
) which then “publishes” them to the rest of the database.
Transactions are, in an abstract way, similar to locks: they allow you avoid issues with other clients messing up what you’re trying to do.
A classic example of a task begging for transactions is transferring money from one bank account to another. You would want to decrement the balance of one account and increment the balance of the other. If the database goes down or the originating account balance goes negative midway, you want to cancel the whole operation. Transactions can make that possible.
However, you can easily shoot yourself in the foot when writing transactions if you keep them running too long.
That’s because once a transaction acquires a lock, it holds onto it until the transaction commits.
For example, let’s say Client 1 opened up psql
and wrote the following:
BEGIN;
SELECT * FROM backpacks WHERE id = 2;
UPDATE backpacks SET content_count = 3 WHERE id = 2;
SELECT count(*) FROM backpacks;
-- ...
Oops! Client 1 has stepped away from their desk:
Someone just brought home some cupcakes!
Even though Client 1 is effectively “done” updating the row with id = 2
, they still have the lock.5
If another client wanted to delete this row, they would run this:
DELETE FROM backpacks WHERE id = 2;
-- ...
-- ?
But it would just hang. It wouldn’t actually delete anything until Client 1 came back and committed the transaction.
You can imagine how this can lead to all kinds of scenarios where clients are holding onto locks for much longer than they need, preventing others from successfully making queries against or updates to the database.
Postgres has a remarkably powerful feature: you can store queryable and efficiently serialized JSON as a value in a row.6 In many ways, it makes Postgres have all the strengths of a document-oriented database (like MongoDB, for example) without having to spin up a new service or coordinate between two different data stores.
However, it has its downsides if used improperly.
While JSONB is quite flexible, Postgres doesn’t keep track of JSONB columns’ statistics which can mean that an equivalent query against a single JSONB column can be significantly slower than against a set of “normal” columns. Here’s a great blog post with someone demonstrating it making things 2000x slower!
A JSONB column can have basically anything in it–that’s one of the main reasons it’s so powerful!
But it also means you have few guarantees about how it’s structured.
With a normal table, you can look up the schema and see what a query will return.
Will a key be written in camelCase? snake_case?
Will states be described with boolean true
/false
? Or with enums like yes
/maybe
/no
?
You have no idea with JSONB as it doesn’t have the same static typing that Postgres data normally has.
Let’s say you have a table called backpacks
with a JSONB column data
where you have a brand
field.
You want to find the JanSport backpacks
because you love the early 90s aesthetic.
So you write the query:7
-- WARNING: Does not work!
select * from backpacks where data['brand'] = 'JanSport';
and you get the error back
ERROR: invalid input syntax for type json
LINE 1: select * from backpacks where data['brand'] = 'JanSport';
^
DETAIL: Token "JanSport" is invalid.
CONTEXT: JSON data, line 1: JanSport
What gives?
Postgres is expecting the right-hand side type of the comparison to match the left-hand side type; that is, for it to be a correctly formatted JSON document–therefore, it needs to be a JSON object, array, string, number, boolean, or null.
Keep in mind that none of these types have anything to do with Postgres types like boolean
or integer
. And NULL
in SQL works very differently from JSONB’s null
, which acts more just like a normal type.8
To correctly write this query, you need to enable Postgres to be able to do some coercion.
select * from backpacks where data['brand'] = '"JanSport"';
-- ^ This is really equivalent to the following (because Postgres knows the left-hand side is `jsonb`)
select * from backpacks where data['brand'] = '"JanSport"'::jsonb;
-- Alternatively, you could convert the left-hand side to Postgres `text`:
select * from backpacks where data->>'brand' = 'JanSport';
Note the double quotes inside the single quotes.
JanSport
on its own isn’t valid JSON.
What’s more is there are a bunch more operators and functions that are specific to JSONB and are hard to remember all at once.
Hope you found this useful.
Thank you to Lillie Chilen, Monica Cowan, Steven Harman, and KWu for encouragement and feedback on this post.
If you have any corrections, feedback, or comments, you can find me on basically all sites as hibachrach
.
If you print it on A4 paper, it’s 3,024 pages; just another reason that standard is better, I guess. ↩
This avoids having to use the more standard COPY
statement which unfortunately often requires escalated privileges which you may or may not have access to. ↩
That is…typically. Some ALTER TABLE
forms (like adding constraints) require a less restrictive lock.
See its page in the docs for more info. ↩ ↩2
It used to be that any default could make adding a column slow, but this was fixed in Postgres 11, which feels recent to me but of course was released 6 years ago. ↩
In this case, the lock at issue is a row lock, which (as you might guess) is a lock that is on the row. Row locks work fairly similarly to the table-level locks we were discussing earlier, but are a bit simpler and only have 4 levels. ↩
Postgres has both JSON values (where the text is stored as text) and JSONB where the JSON is converted to an efficient binary format. JSONB has a number of advantages (e.g. you can index it!) to the point where one can consider the JSON format to just be for special cases (in my experience, anyway). ↩
This is leveraging the relatively new subscripting syntax introduced in Postgres 14 (in my opinion, better than the original ->
syntax which feels less familiar compared with how JSON is traversed elsewhere). ↩
Most notably, 'null'::jsonb = 'null'::jsonb
is true
whereas NULL = NULL
is NULL
↩
2020-10-07 15:00:00
In this article I go over three things that, in my mind, would make JavaScript better. None are new ideas. This post is an expansion of a tweet I had when I saw someone asking about improvements for JS. (though probably are impossible for various reasons). I’m going to be primarily speaking about browsers and the web, though much of this might apply to Node.js (though I’m not as familiar with that area so I can’t speak on it confidently).
Just as a heads up: I’m not really talking about different ECMAScript versions (e.g. ES6, ES2019, etc.) here–I’m talking about how most programming languages refer to versions.
Right now, there are two versions of JS: Strict mode and “sloppy mode”. Feature detection is done dynamically: if a script makes use of any feature or change in the language that isn’t supported by the environment running it, it will error, either silently or loudly depending on what that feature is.
To get around this, developers do one or both of the following:
To get around browsers not supporting features, we use tools like Babel to convert JS making use of newer features to JS supported by all or polyfill them with. However, this has a few problems:
browserslist
reduce the impact of this point,
but they require upkeep. If a website stops development now but stays online,
its JS bundle won’t get any faster despite the herd of browsers moving to
support the newer features in the source JS.The idea is that you can identify what a browser might need based on the version presented in its “user agent” (UA). There’s a whole article on MDN on why this is a bad idea in general. However, this hasn’t stopped influential companies like Twitter from doing it.
Google Developers instead encourages using the support for <script
type="module">
as a discriminating factor. This seems a bit
better, but of course this is just one test–Safari is not an evergreen
browser and so despite it supporting modules, we can’t
rely on this to check for support for “generally new feature” availability in
the medium or long-term.
As I said at the beginning, there already is a versioning scheme for JS. Strict mode changes the behavior of JS scripts in a backwards incompatible way: if you had a script that worked in “sloppy mode”, it might break in strict mode.
However, it doesn’t look like there are any plans to further extend this approach. When “#SmooshGate” (an incident of browsers accidentally breaking sites relying on old JS extensions by adding incompatible features) happened, versioning was suggested by more than one person. After all, with versioned JS, the issue evaporates. Commenters on Hacker News responded to these suggestions, suggesting that supporting multiple distinct versions introduces significant complexity for developers of JS engines. One person even noted
This has been discussed at length and they have decided not to do it. It’s not a missing feature, it’s by design.
There are other negatives to versioning expounded on in this wonderful article, such as the following quoted here:
- Engines become bloated, because they need to implement the semantics of all versions. The same applies to tools analyzing the language (e.g. style checkers such as JSLint).
- Programmers need to remember how the versions differ.
- Code becomes harder to refactor, because you need to take versions into consideration when you move pieces of code.
I can’t speak much to the work of maintaining engines–this is done by engineers far more skilled than myself. My immediate reaction is that managing different versions might enable stricter handling of various code, leading to simplification, though that’s probably a naive perspective.
On the topic of remembering how versions differ, I would say this is simple in comparison to the inconsistent mess of browser compatibility, JS transpilation configuration, and generally frequent change within the ecosystem (though I will be the first to say that the last point has been fairly exaggerated). In other languages, versions change, and this is considered business as usual.
With regards to added difficulty in refactoring, I would say that this again is probably simpler than other things which we do semi-regularly, such as upgrading major versions of important libraries (e.g. jQuery, webpack), and is likely able to be automated. Additionally, the difficulty is highly dependent on the audacity of those at the reins of JavaScript, who, based on the current environment, seem unlikely to cause unnecessary upset.
The main area where I wish this were the standard within JS is with if
/else
,
try
/catch
, and switch
/case
statements. This is something that I use very
frequently within Ruby
if
/else
const a =
if (cond) {
b
} else {
c
};
try
/catch
const a =
try {
somethingThatMightFail()
} else {
fallbackValue
};
switch
/case
const a =
switch (b) {
case 'apple' {
'fruit'
}
case 'broccoli' {
'vegetable'
}
default {
'unknown'
}
};
Its possible this would need to use different keywords to replace case
and
default
for the sake of JS interpreters and maximizing backwards
compatibility because case
s function as labels.
To achieve this, do
expressions were proposed 2 years
ago, which satisfy the requirements with slightly
more verbose syntax. E.g. for the if
/else
, you’d write
const a = do {
if (cond) {
b
} else {
c
}
};
However, the proposal is still at stage 1 of the 4-stage TC39 (effective JS
steering committee) process, though it’s still being
discussed. Some have asked “why do we need the do
?”
and make the first syntax (without the explicit do
) part of the language,
though this can’t be done without interfering with existing uncommon language
features (another example of not being able to change syntax due
to “version constraint”)
I would explain this, but there are actually people already solving this problem, and they’ve put together this article explaining how caching can be improved around the web.
However, there are still unresolved issues here: how does this work for different browser targets? It’s fine if all libraries are only using browser-supported features, but we all know that that won’t be consistent for all features across all browsers into the future. A lot of this builds on the issues presented in the section on versioning above: if there are no ways to talk consistently about versioning, then it’s much harder to solve these problems in an automated way.
2020-04-24 15:00:00
“Where should this information live?” is a common question I encounter daily as a software engineer. I (alone or in consultation with others) will make a decision and that information needs to persist…somewhere. The goal of course is to prevent some future engineer (often myself) from encountering some code and asking “why?” or, even worse, “what?”.
I’ve found there’s a sort of hierarchy to the accessibility of various information that I found useful in sharing.
In order of most accessible to least, the hierarchy is as follows:
This list may be different for other people and may change in time as tools get better, but for now, this has held true for me.
I’m sure that similar things have also been said before. Shoulders of giants and all that—would love to read any previous writings on similar topics!
So! Let’s get into some of the reasoning…
By far, the most important place to put information is in the code itself. Why?
grep
, ag
, ripgrep
, or your
favorite file searcherBecause of their inherent visibility, identifiers1 should always be the first place to put information, especially
I’m going to intentionally give you no context for the following code:
if (
userSession['signed']['admin']['update'] ||
AdminOverrides[userSession['signed']['id']]
) {
repository.delete();
return render({
status: 200,
body: {
notice: 'Repository successfully deleted.'
}
});
} else {
return render({
status: 403,
body: {
errors: ['You are not allowed to perform that action.']
}
});
}
Okay, pretty straightforward. But consider this version instead:
const userHasUpdateAccess = (
userSession['signed']['admin']['update'] ||
AdminOverrides.update[userSession['signed']['id']]
);
if (userHasAdminUpdateAccess) {
repository.delete();
return render({
status: STATUS_CODES.ok,
body: {
notice: 'Repository successfully deleted.'
}
});
} else {
return render({
status: STATUS_CODES.unauthorized,
body: {
errors: ['You are not allowed to perform that action.']
}
});
}
We’ve made a few updates. First, we’ve pulled the condition out into its own variable. What that’s done is document the intention of what the condition is supposed to mean. This has a few benefits:
if
may get much larger. It’s always easier to keep things tidy one
piece at a time (as opposed to going back and cleaning it up later)!The other two changes are to replace the HTTP status codes (which are
essentially magic numbers) with references to a global
STATUS_CODES
object. While HTTP status codes are fairly well known (e.g. a
“404” is meaningful to many that have never done web development), they’re
likely not going to be better known than English.
Generally, breaking things down like this makes the answer to “why?” very obvious.
Unlike libraries which can often be worked around, specific hard-to-parse features of a language’s syntax may be unavoidable. For example, most languages have regular expressions which are infamous for how easy they are to mess up. Consider the following regular expression in JavaScript:
const IS_VALID_EMAIL_REGEX = /^[^@]+@[^@]+$/;
There’s not really much more we can do in terms of adding/changing identifiers to clarify how the regular expression determines which email addresses are valid and which aren’t–you just have to know the specific syntax of (JavaScript) regular expressions. In cases like these, an in-code comment (which we’ll talk about more in the next section) is the next best place to put this information:
// Matches one or more non-`@` characters, folloed by a `@` character, followed
// by one or more non-`@` characters.
const IS_VALID_EMAIL_REGEX = /^[^@]+@[^@]+$/;
One of the points made about code also apply to in-code comments: they are easily searchable using a file searcher. Also, they do not need any further seeking out than code,4 unlike something like commit messages.
However, as they are in a non-programming language, they can be a bit less predictable in terms of structure—you’ll have a harder of a time breaking an English sentence into an abstract syntax tree than compared with something like JavaScript!
Additionally, in-code comments are related to their subjects typically by proximity alone. If someone moves that code without taking the comment with it, the comment ceases to be helpful (or, worse, becomes confusing/misleading). We talk about this more in the “Pitfall” section below.
Comments, while still being very visible, afford a much greater level of flexibility in comparison with code identifiers for documenting behavior, decisions, etc. Comments are where you put
For example, say we have this method:
// Must be done in reverse order to be done in O(N) time due to
// Kleppner's Law of Stupid Data Structures
function smooshify(newElements) {
for (let i = newElements.length - 1; i >= 0; i--) {
this.flarbo.insert(newElements[i])
}
}
At some later point, it turns out are some null
elements that need to be
screened out:7
// Must be done in reverse order to be done in O(N) time due to
// Kleppner's Law of Stupid Data Structures
function smooshify(newElements) {
const nonNullNewElements = newElements.withoutNulls();
for (let i = newElements.length - 1; i >= 0; i--) {
this.flarbo.insert(nonNullNewElements[i])
}
}
With the code as it is now, another dev may stumble along one day and ask “Is
withoutNulls
iterating in reverse order?” While this may be a bit of a trivial
example, these things can lead to a lot more headache down the line
The only way to truly avoid mistakes like this is by is by keeping a vigilant eye on comments. Some techniques I’ve found to be helpful:
Keep comments as closely positioned to the code they’re describing as
possible. E.g. if talking about code in the first branch of an if
statement,
put that comment in the first branch, not above the entire statement:
// comment about why `floarp` should be used --- BAD
if (somethingOrOther()) {
// comment about why `floarp` should be used --- GOOD
floarp();
} else {
blarg();
}
Make references to specific identifiers in comments as these tend to garner more attention and help establish a more explicit relationship between the comment and the code
Most code that I’ve interacted with is version-controlled with something like Git, and if not Git, some other version-control software like Mercurial. I’m going to be referring to Git exclusively in this section, but I’m sure that there are similar tools for other version-control software.
Unlike code or in-code comments, commit messages are not immediately visible
when browsing a codebase. However, they can be made much more easily accessible
with tools like Fugitive (for Vim) or
GitLens
(for VSCode). However, they can be searched–just run git log
and press
/ and BAM! You can search through all your commit history (thanks
to less
)
Unfortunately, commit messages can sometimes be squashed or lost during a rebase, so the information placed there may not be preserved forever, though this mostly depends upon the other collaborators working on the repository.
Much has been written about how to write good commit messages, so I won’t be too comprehensive here. However, I’d say the things I’m talking about here are, in my opinion, the most important things about writing a useful commit message. As much as I like consistency, the grammatical aspects of commit message guidelines are not nearly as helpful as guidelines on the content of the message (as long as it’s readable!).
Commit messages are all about change. A commit itself is an object representing a change to a codebase and commit messages ought to do the following:
These are much easier to accomplish when the commits you make are small and focused (within reason).8
While commit messages allow you to tell a story, any single commit rarely allows
you to tell the whole story. If referencing earlier changes, I highly encourage
you to make such references by using the commit hash. This will enable those
reading those commit messages later to gain the same context you had when
writing the message in the first place with a simple git show
. A word of
warning however: commit hashes will change when you rebase or amend a commit, so
if referencing an earlier commit in a commit message, make sure to update that
hash if the earlier commit is ever altered.
These are your GitHubs, your Asanas, your Jiras. I’m lumping issue trackers and hosted version control together because some of these services function as both (e.g. GitHub).
While there’s a large amount of variance here, I always have found messages and comments to get lost somehow, even when generally good search functionality is present. Some services are better than others at this, but even in GitHub (which automatically doubly-link issues to pull requests), comments will be collapsed or hidden in large pull requests, making them a pain to excavate.
These services are all about facilitating conversation and communication. As such, their responsibilities are focused on those topics:9
Hope you found this helpful in answering the question “where do I put this information?”. If you have any thoughts, I’m on Twitter.
Identifiers include variable names, function names, class names/types, operators—basically anything that has a name that you can control! ↩
I’ll probably make another post about this, but I’ve found
answering the question of “when” to be a bit of a slippery slope, especially
in function names. When you answer the question of “when”, you are often doing
so at the expense of “what”. Who hasn’t written a function like onClick
or
after_save
that spans 20+ lines, accomplishing many disparate tasks? One
technique I’ve employed when these are unavoidable (as these identifiers are
often dictated by some external API) is to push all sense of “how” out of the
bodies of these functions—they merely list off the things to do and the “how”
is pushed into other functions. If you read on, we’re trying to lighten
cognitive load here! ↩
However, they may be slightly dimmed depending on the code editor. ↩
I most often use this for adaptations of code from StackOverflow or blog posts. ↩
If possible, these should be replaced with good identifier naming, type checking, validations, etc. ↩
If this were real code, I would hope that someone would also leave a comment explaining why we are doing this screening! ↩
I found this article a great summary of the how/why of keeping commits small ↩
Of course, these services have usages beyond the topics addressed in this post like:
2018-04-01 15:00:00
Ergonomics often comes up when talking with HR during onboarding: it’s the reason why you should use comfortable, supportive chairs, keyboards, mice, and such. “Battle scars” from work shouldn’t be encouraged. If your eyes, wrists, or butt cheeks are hurting, try to address that problem. You’re doing labor. Labor that most likely isn’t as tolling, as say, construction, but labor nonetheless. Take care of your body. If you try to fight it, it’s likely to win. Even if you’re not feeling active pain now, you might start experiencing symptoms after years of damage. “It works for me” shouldn’t be the end of it. Get an ergonomic chair, keyboard, and mouse to reduce strain. Here12 are some good guidelines on reducing eyestrain (I can personally vouch for tools like f.lux!).
What does this mean for software? If you don’t like the way you’re interacting with your tools (physical or virtual), you’re not going to have as much fun using them.
Sometimes that can be a good thing: if a tool shouldn’t be used, then you can design your system with that idea in mind. As we are all designers in some way, this should be a familiar idea—e.g. if you add a feature that forces a user to do 10 clicks instead of one, they’re less likely to use that feature (c.f. trying to close your Amazon account).
Notice that I’m not talking about speed. The amount of time that you save with one configuration or another probably isn’t going to save you the amount of time that you put into setting it up (unless something was really wrong before-hand). But reducing UX friction can often make you more productive, reduce extraneous repetitive motion, and eliminate unnecessary mental work to get from point A to point B. Some things that help towards this end include:
Everything that you do to configure your machine is most likely increasing the barrier to entry somehow. That’s okay. It’s your machine—it should be most comfortable for you. However, that doesn’t mean there won’t be a time where someone else might have to use your keyboard or read your monitor. It should be easy enough for that visitor to understand what’s going on or do some simple tasks without you restarting your damn machine. Note that that doesn’t mean disabling all of your config when someone walks over4.
Your tools or configuration won’t make you a significantly better developer. It might even make you a more annoying developer if you won’t stop talking about such topics. And it always takes time to customize these things—the increased efficiency won’t make up for that. Again, it’s about getting you to enjoy using the tools while reducing strain, whether that strain is physical or mental.
I use tmux inside of iTerm25. Tmux is a “terminal multiplexer” which is an incredibly geeky way of saying it allows you to have windows/splits inside of a OS level window. Here are my main motivations for using it.
When you start up tmux you’re starting up a server, on which you create and attach “sessions” (basically just workspaces) where you can create windows (really closer to tabs) which can have splits. These sessions can be attached to and detached from at will. And that action is a completely distinct one from opening or closing iTerm2, meaning if iTerm2 has to close for one reason or another, my windows and splits set up exactly as when I left them.
This is great on a local machine but invaluable on remote sessions: when I was doing research on machines that could only be SSH’d into, a random drop in the Internet connection just meant I had to reattach to the session that was running on the remote server. Note that the server will still die if your computer has to restart.
I also find that this navigation is much easier than fiddling with actual windows in a GUI. Which tend to be pretty poorly organized for search and retrieval. Even in macOS, if I have a window that is minimized, I have to either use my mouse to inspect my collection of hidden windows or cycle through them using control+f46.
This is going to be a bit of a recurring theme, but tmux allows you to go from split to split, window to window, or session to session without pressing many keys simultaneously. Instead, you can define a common prefix (I have control+space) and then press the tmux-specific shortcut of your choosing.
If you’re on a Mac, you can use Homebrew:
brew install tmux
Here’s a tutorial to get you started.
I could talk probably all day about Vim and why I love it. I’ll try to summarize here to save your time and/or sanity. I will note that Vim has a fairly steep learning curve but I find is worth it in the end. See here for a great blogpost about Vim and usability.
Vi or Vim is on basically every *nix machine that you might come across. If you know how to use it in one place, you can use it anywhere.
Every interaction with Vim is intended to be done with the keyboard. There are some things that can be done with the mouse (like scrolling or cursor movement), but it’s primarily designed to be manipulated via the keyboard. That makes it generally quite fast, eliminating the time to move your hand to the mouse and back.
This is probably the biggest and most Vim-specific point here. When you manipulate text in Vim, you do so in a sort of “language”, complete with verbs, nouns, adjectives, adverbs, etc.
Let’s say I’m editing a Ruby script and I’ve got this double negative here. We
want to make this a bit more readable, replacing the unless
with an if
and
the empty?
with a present?
.
Here are all the key presses I needed to do to perform the above:
ciwifjjwwwct?presentjj:wq
Let’s break that down:
ciw
: This is basically a “sentence”. I’m pressing these keys them in what is
called “normal mode”. c
means “change”, i
means “inner”, and w
means
“word”. “Change” means delete and then go into “insert mode”, the mode where
when I type, those characters appear on screen. I applied that “verb” to what
is called a “text object”. Here that “text object” is made up of the adjective
“inner” with the noun “word”.if
: This is just me typing the word “if”.jj
: This is a command I have set up for “insert mode” that gets me back to
“normal mode”. This is escape by default but I find this much
easier.www
: Here I’m pressing that “noun” w
three times. Since there’s no “verb”
here, that just moves my cursor 1 “word” each time. I could have also typed
3w
and achieved the same thing.ct?
: Here I’m using the same c
verb but this time with the “noun” t?
which means “to ?
”, i.e. all the characters from my current position up to
(but not including) the next incidence of the character ?
on the current
line.present
: This is just me typing the word “present”.jj
: Same as before.:wq
: Here I’m performing what is known as a “command”. :
gets me into
“command mode” (moving my cursor temporarily to the bottom of the Vim window)
where I then type w
for “write” (meaning “save”) and q
for “quit”.This is a simple example, but as you can see, you can mix and match all these
building blocks to manipulate your files in quite complex ways. Many Vim users
also create plugins that add core “parts of speech” to the editor. E.g.
vim-surround enables the sequence cs"'
which allows you to
“change the surrounding characters” (cs
) from "
to '
as well as the
sequence ysiw]
“add surrounding characters” (ys
) to the “inner word” (iw
),
specifically brackets (]
).
(Partly thanks to tmux) I can run Vim in the same program that has my
foreman
procs, my git interface, my logs, and pretty much everything else.
Switching between all of those things is super easy thanks to a plugin I use
called vim-tmux-navigator. Basically I can use
to navigate between tmux and Vim panes indiscriminately.
When I need to run a test, I have a plugin called vim-test that allows me to run a test in a neighboring tmux pane and then interact with that terminal session afterwards.
There are many other niceties here as well (e.g. sending the contents of a SQL
query edited in a Vim session to an interactive psql
session) that I may go
into in a later post.
Again, I can talk about Vim all day (and probably will). There are many things
not covered here that I will return to later (possibly including a breakdown of
my .vimrc
).
Like I said before, Vim is famous for having a bit of a steep learning curve. I’d recommend trying it out in your leisure time because it will slow you down at first.
Once again, the download for the newest version (probably more up to date than
what you have on your machine already) is on Homebrew as well. However, I’d
recommend installing the relatively recent fork neovim
instead (it has a few
nicities including better built-in tutorial:
brew install neovim
Launch it from the shell (using the name nvim
, not neovim
). Then I’d
follow the built-in tutorial by typing :Tutor
. The first online resource given
at the bottom of the tutorial, “Learn Vim
Progressively” is also great—it’s how I first
started.
FZF is a fuzzy-finder for the shell that you can use in a bunch of different ways, like…
.gitignore
)7
I couldn’t manage to record this one without any potentially sensitive information, but it works as you might expect.
At its core, FZF is just a way of filtering data in a human-friendly way. You
can pipe anything in and get the same fuzzy-find interface
(e.g. cat file.txt | fzf
).
Once again, if you can, just use Homebrew.
brew install fzf
See the project’s installation instructions for details, including howt to integrate FZF with tmux and vim. These integrations work incredibly well so I’d highly recommend setting that up.
That’s it for now! More posts on this stuff to come, I’m sure.
https://ergonomics.ucla.edu/injuries-and-prevention/eye-strain.html ↩
This isn’t a cure-all for repetitive strain injuries. Stretching and regular breaks are more important. ↩
It can be a great way to learn tricks when you see someone do some keyboard wizardry and you ask “How did you do that!?” ↩
The main reason for using it over the default macOS Terminal is for 256 color support. ↩
Just as a heads up: you can cycle through open applications with ⌘+tab and open windows for your current application with ⌘+`. ↩
I’m not sure why the colors on this recording are inaccurate, but I assure you Vim on my machine is quite colorful. I use the colorscheme seoul256. ↩