What actually happens to an IP address in an analytics pipeline

Every analytics vendor has a sentence about IP addresses. They sound similar and they are not. The differences only become visible when you ask what is written to disk, and in what order.

Four phrases, four different systems

Anonymised after collection means the address was received, stored, and then processed. There is a window, sometimes a long one, where the original sits in a table or a log. Whatever the retention policy says, it existed and it was written down.

Truncated means part of the address is discarded, usually the last octet. This is better than nothing and much weaker than it sounds: a truncated IPv4 address still identifies a network of 256 addresses, which in a rural area or a small office is a household or a company.

Hashed means the address was run through a hash function. On its own this is close to meaningless, because the entire IPv4 space is four billion values and a modern machine can hash all of them in seconds. An unsalted hash of an IP address is a lookup table away from being the address.

Salted and rotated means the address is combined with a secret that changes, and the secret is destroyed on a schedule. Once the salt is gone the identifiers cannot be reversed even with the full database, because the material needed to rebuild the table no longer exists.

What Metricward does, in order

The pipeline for a single event, exactly as it runs:

• ·The ingestion service reads the connection address from the socket. It exists as an argument in a function, in memory.
• ·It is used for a geographic lookup against a MaxMind database file held locally, so the address never leaves the process for a third-party lookup service.
• ·It is combined with a per-project salt, the user agent and the accept-language header, and hashed. The result is a 22 character opaque string.
• ·The address goes out of scope. It is not stored, not logged, not attached to a trace, and there is no column in any table that could hold it.
• ·The salt lives only in a Redis instance with persistence disabled, expires 24 hours after the project's local midnight, and is destroyed when it expires.

The parts that are easy to get wrong

Every one of these is a mistake we either made and fixed, or specifically designed against, and each one silently undoes the guarantee above.

• ·The web framework's default request logger writes the remote address on every request. We shipped that for exactly one commit, and it wrote visitor IPs to disk thousands of times a second while the architecture diagram said otherwise. There is now a test that fails if an IP-shaped string appears in the ingestion log output.
• ·The salt Redis had persistence enabled, so the salt reached an append-only file and would have survived in backups. It now runs on a separate instance with both AOF and snapshots off, because the stream needs durability and the salts need the opposite.
• ·An error tracker attached to the ingestion service will happily capture the whole request, including the address, and send it somewhere else entirely.
• ·The query string carries more identifying data than the IP most of the time. An allowlist drops everything except campaign parameters before storage, because a blocklist eventually misses the one your framework invented last week.

How to check any vendor, including us

Ask for the field list of the table the events are stored in. Not a policy page, not a summary, the columns. If a column exists that can hold an address, then the address can be in it, whatever the retention policy says today and whoever owns the company next year.

The strongest version of this promise is not a rule about deleting data. It is having nowhere to put it.