Attestation compatibility guide

- -

Apps using the Play Integrity API or - obsolete - SafetyNet Attestation API to check the authenticity/integrity of the OS can support - GrapheneOS by using the standard Android hardware attestation API instead and - permitting our official release signing keys. Android's - hardware - attestation API provides a much stronger form of attestation than the Play - Integrity API with the ability to whitelist the keys of alternate operating systems. - It also avoids an unnecessary dependency on Google Play services and Google's - Play Integrity servers.

- -

The standard hardware attestation API can be used to verify the authenticity/integrity - of the hardware, firmware, OS and the app running on it. It provides a verified boot key - fingerprint for the OS for permitting secure aftermarket operating systems. The app id, - signing key fingerprint(s) and version code of the app enabling hardware attestation are - included in the signed public key certificate for the generated key. This enables the - app's service to make sure the app is genuine and unmodified along with chaining trust - through the OS to the app which can sign messages with the attested hardware keystore - key to prove they come from their app running on top of a verified OS, firmware and - hardware. The only practical way to bypass hardware attestation is through exploiting - the hardware keystore to obtain attestation signing keys, which is protected against by - the ability to revoke keys that are being misused. Play Integrity API strong integrity - level is directly based on the hardware key attestation API, but apps using it directly - can support aftermarket operating systems, check the hardware attested OS patch level - and other provided information. The hardware attestation API also supports pinning-based - security instead of only root-based security where keys can be leaked and used to fake - attestations. Apps can use pinning to establish a much higher security pairing with a - specific device to obtain fresh attestations with a very high level security based on - the security of the device's own hardware keystore rather than the overall ecosystem. - Hardware attestation also doesn't require using any Google service beyond regularly - fetching the list of revoked keys for root-based attestation. The app's service doesn't - have to go down or start permitting anything if the Google services becomes unavailable - or blocks the app from using it for one reason or another. Using hardware attestation is - therefore more reliable and lower risk for apps.

- -

Devices have been required to ship with hardware attestation support since Android - 8. You can use hardware attestation on devices running Android 8 or later when the - ro.product.first_api_level system property isn't set to 25 or below, - which indicates they launched with Android 8 or later with hardware attestation - support as a mandatory feature. On older devices, you can continue using the Play - Integrity API. Some low quality devices shipped broken implementations of hardware - attestation despite the requirement to have it working for CDD/CTS certification and - the Play Integrity API currently still passes on those devices wrongly claiming them - to be CTS certified. If you don't want to fail on those devices, then you can start - with hardware attestation and fall back to the Play Integrity API or do both and - accept either passing as success.

- -

Google provides a key - attestation library with examples. Our MIT - / Apache 2 licensed Auditor app can be used as a reference implementation for - verifying hardware-based attestations. There are some subtleties in the verification - process such as making sure only the 2nd certificate in the chain (the one signing the - certificate for the key generated by your app) has an attestation extension to prevent - making a fake attestation by extending the chain. You can reuse our code and simply - omit support for an app generated attestation signing key (attest key) and the other - pinning support.

- -

After verifying the signature of the attestation certificate chain and extracting - the attestation metadata, you can enforce that verifiedBootState is - either Verified or SelfSigned. For the - SelfSigned case, you can check that verifiedBootKey matches - one of the official GrapheneOS verified boot keys. These are the base16-encoded - verified boot key fingerprints for the official GrapheneOS releases:

- -

0508de44ee00bfb49ece32c418af1896391abde0f05b64f41bc9a2dfb589445b: Pixel 9a
af4d2c6e62be0fec54f0271b9776ff061dd8392d9f51cf6ab1551d346679e24c: Pixel 9 Pro Fold
55d3c2323db91bb91f20d38d015e85112d038f6b6b5738fe352c1a80dba57023: Pixel 9 Pro XL
f729cab861da1b83fdfab402fc9480758f2ae78ee0b61c1f2137dd1ab7076e86: Pixel 9 Pro
9e6a8f3e0d761a780179f93acd5721ba1ab7c8c537c7761073c0a754b0e932de: Pixel 9
096b8bd6d44527a24ac1564b308839f67e78202185cbff9cfdcb10e63250bc5e: Pixel 8a
896db2d09d84e1d6bb747002b8a114950b946e5825772a9d48ba7eb01d118c1c: Pixel 8 Pro
cd7479653aa88208f9f03034810ef9b7b0af8a9d41e2000e458ac403a2acb233: Pixel 8
ee0c9dfef6f55a878538b0dbf7e78e3bc3f1a13c8c44839b095fe26dd5fe2842: Pixel Fold
94df136e6c6aa08dc26580af46f36419b5f9baf46039db076f5295b91aaff230: Pixel Tablet
508d75dea10c5cbc3e7632260fc0b59f6055a8a49dd84e693b6d8899edbb01e4: Pixel 7a
bc1c0dd95664604382bb888412026422742eb333071ea0b2d19036217d49182f: Pixel 7 Pro
3efe5392be3ac38afb894d13de639e521675e62571a8a9b3ef9fc8c44fd17fa1: Pixel 7
08c860350a9600692d10c8512f7b8e80707757468e8fbfeea2a870c0a83d6031: Pixel 6a
439b76524d94c40652ce1bf0d8243773c634d2f99ba3160d8d02aa5e29ff925c: Pixel 6 Pro
f0a890375d1405e62ebfd87e8d3f475f948ef031bbf9ddd516d5f600a23677e8: Pixel 6

- -

The verifiedBootKey field is binary data so you either need to encode - it as base16 to compare with these or convert these to binary. An easy approach is - storing the permitted key fingerprints in a set and enforcing that the verified boot - key is in the permitted set when verifiedBootState is - SelfSigned.

- -

GrapheneOS regularly adds support for new devices so you should have a process for - regularly adding the new verified boot key fingerprints from this page.

- -

The hardware attestation API also provides other useful information signed by the - hardware including the OS patch level, in a way that even an attacker exploiting the - OS after boot to gain root cannot trivially bypass. It's a better feature than the - Play Integrity API which has to be designed for the lowest common denominator.

- -

GrapheneOS users are strongly encouraged to share this documentation with app - developers enforcing only being able to use the stock OS. Send an email to the - developers and leave a review of the app with a link to this information. Share it - with other users and create pressure to support GrapheneOS rather than locking users - into the stock OS without a valid security reason. GrapheneOS not only upholds the - app security model but substantially reinforces it, so it cannot be justified with - reasoning based on security, anti-fraud, etc.

- -

Apps banning GrapheneOS

- -

This is a list of the apps banning GrapheneOS with the Play Integrity API with - links to their Play Store pages for leaving feedback:

- -

myGov (Australian government app)
gov.br (Brazilian government app)
Ticketcorner
Authy
eBay
McDonald's (International app used for many but not all countries not including the US)
Dott
Swissquote
SwissID
TK-App (German health insurance app which uses it for fingerprint login)
IO (Italian government app which uses it for the digital wallet feature)

- -

In addition to leaving feedback for these apps on the Play Store, file support - requests and leave feedback on third party review sites. Ask them to stop banning - GrapheneOS and explain that it's a much more secure OS than what they permit which - does not lose any of the standard security model. Explain that they can use the - hardware key attestation API to verify that a device is running GrapheneOS to permit - it alongside an OS licensing Google apps as they do with the Play Integrity API - already. Make sure to push back against false claims that it has something to do - with compatibility or security issues. The only reason they aren't permitting it is - because we do not license Google Mobile Services (GMS) and these apps are enforcing - Google's business interests rather than security.

GrapheneOS servers

- -

This is a detailed list of the public GrapheneOS servers.

- -

We use hardened local machines for building and signing rather than servers outside - our physical control, so information on that infrastructure is outside the scope of this - page but may be provided in the future elsewhere.

- - - -

GrapheneOS website

- -

Repository
Issue tracker

- -

Specs:

- -

3x OVH VPS vps2023-le-2
2 core
2 GB memory
40 GB NVMe SSD storage
500 Mbit/s bandwidth

- -

1x BuyVM Slice 1024
1 core
1 GB memory
20 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

grapheneos.org
www.grapheneos.org
grapheneos.app
www.grapheneos.app
grapheneos.ca
www.grapheneos.ca
grapheneos.com
www.grapheneos.com
grapheneos.dev
www.grapheneos.dev
grapheneos.foundation
www.grapheneos.foundation
grapheneos.info
www.grapheneos.info
grapheneos.net
www.grapheneos.net
grapheneos.ovh
www.grapheneos.ovh
grapheneos.page
www.grapheneos.page
vanadium.app
www.vanadium.app

- -

IPs:

- -

51.222.156.101 (0.grapheneos.org) — OVH bhs6
2607:5300:205:200::29c6 (0.grapheneos.org) — OVH bhs6
209.141.35.164 (1.grapheneos.org) — BuyVM Las Vegas
2605:6400:20:1131:8088:e08:84e6:632 (1.grapheneos.org) — BuyVM Las Vegas
54.37.41.189 (2.grapheneos.org) — OVH gra8
2001:41d0:304:200::b109 (2.grapheneos.org) — OVH gra8
51.79.160.50 (3.grapheneos.org) — OVH sgp2
2402:1f00:8000:800::16d6 (3.grapheneos.org) — OVH sgp2

- -

Ports:

- -

TCP 22 ssh
TCP 80 http
TCP 443 https

- -

Staging GrapheneOS website

- -

Repository
Issue tracker

- -

Specs:

- -

BuyVM Slice 1024
1 core
1 GB memory
20 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

staging.grapheneos.org

- -

IPs:

- -

199.195.250.78 — BuyVM New York
2605:6400:10:9d6:6d84:e183:acda:16d7 — BuyVM New York

- -

Ports:

- -

TCP 22 ssh
TCP 80 http
TCP 443 https

- -

GrapheneOS release servers

- -

These are the static file servers for GrapheneOS releases and our app - repository. These are used by the releases page and web installer along with the - System Updater and App Store (app repository client) within the OS.

- -

Repository
Issue tracker

- -

Specs:

- -

Macarne dedicated server (sponsored by Macarne)
Ryzen 9950X
128 GB DDR5
2x 2 TB NVMe SSD storage
25000 Mbit/s bandwidth

- -

2x ReliableSite dedicated server (sponsored by ReliableSite)
Ryzen 9900X
192 GB DDR5
2x 4 TB NVMe SSD storage
10000 Mbit/s bandwidth

- -

Domains:

- -

releases.grapheneos.org
seamlessupdate.app
www.seamlessupdate.app
apps.grapheneos.org

- -

IPs:

- -

45.90.185.33 (4.releases.grapheneos.org) — Macarne Amsterdam
2a14:3f87:6920:250::100 (4.releases.grapheneos.org) — Macarne Amsterdam
172.96.172.37 (5.releases.grapheneos.org) — ReliableSite Miami
2605:9880:400:1100:15:1240:515:6e (5.releases.grapheneos.org) — ReliableSite Miami
104.194.8.203 (6.releases.grapheneos.org) — ReliableSite Los Angeles
2605:9880:200:20::113 (6.releases.grapheneos.org) — ReliableSite Los Angeles

- -

Ports:

- -

TCP 22 ssh
TCP 80 http
TCP 443 https

- -

GrapheneOS network servers

- -

These are the default servers used by GrapheneOS for connectivity checks, - secure network time, attestation key provisioning and Predicted Satellite Data - Service (PSDS). These either serve empty responses or provide reverse proxies to - other services.

- -

Repository
Issue tracker

- -

Specs:

- -

3x OVH VPS vps2023-le-2
2 core
2 GB memory
40 GB NVMe SSD storage
500 Mbit/s bandwidth

- -

1x BuyVM Slice 1024
1 core
1 GB memory
20 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

grapheneos.network - HTTP/HTTPS connectivity checks
connectivitycheck.grapheneos.network - HTTP/HTTPS connectivity checks
www.grapheneos.network
grapheneos.online - HTTP/HTTPS connectivity checks
connectivitycheck.grapheneos.online - HTTP/HTTPS connectivity checks
www.grapheneos.online
broadcom.psds.grapheneos.org - HTTPS Broadcom PSDS data cache
samsung.psds.grapheneos.org - HTTPS Samsung PSDS data cache
qualcomm.psds.grapheneos.org - HTTPS Qualcomm PSDS data cache
remoteprovisioning.grapheneos.org - HTTPS reverse proxy to remoteprovisioning.google.com
widevineprovisioning.grapheneos.org - HTTPS reverse proxy for Widevine provisioning
time.grapheneos.org - HTTPS time server with millisecond precision X-Time header
supl.grapheneos.org - TLS reverse proxy to supl.google.com
nominatim.grapheneos.org - HTTPS reverse proxy to nominatim.openstreetmap.org, which will become our own instance of Nominatim instead of a proxy
gs-loc.apple.grapheneos.org - HTTPS reverse proxy to Apple's network location service, which will remain an option after we have our own location service
update.vanadium.app - HTTPS reverse proxy to update.googleapis.com for Chromium component updates (will be hosted directly in the future)
dl.vanadium.app - HTTPS reverse proxy to CDNs for Chromium component updates (will be hosted directly in the future)

- -

IPs:

- -

51.222.159.116 (0.grapheneos.network) — OVH bhs6
2607:5300:205:200::2584 (0.grapheneos.network) — OVH bhs6
209.141.37.35 (1.grapheneos.network) — BuyVM Las Vegas
2605:6400:20:387:72d4:dab9:a369:f351 (1.grapheneos.network) — BuyVM Las Vegas
54.37.41.188 (2.grapheneos.network) — OVH gra8
2001:41d0:304:200::902f (2.grapheneos.network) — OVH gra8
51.79.161.36 (3.grapheneos.network) — OVH sgp2
2402:1f00:8000:800::1949 (3.grapheneos.network) — OVH sgp2

- -

Ports:

- -

TCP 22 ssh
TCP 80 http
UDP 123 ntp
TCP 443 https
TCP 7275 supl

- -

GrapheneOS mail server

- -

Repository
Issue tracker

- -

Specs:

- -

OVH VPS vps2023-le-2
2 core
2 GB memory
40 GB NVMe SSD storage
500 Mbit/s bandwidth

- -

Domains:

- -

mail.grapheneos.net
mail.grapheneos.org
mta-sts.attestation.app
mta-sts.discuss.grapheneos.org
mta-sts.grapheneos.app
mta-sts.grapheneos.ca
mta-sts.grapheneos.com
mta-sts.grapheneos.dev
mta-sts.grapheneos.foundation
mta-sts.grapheneos.info
mta-sts.grapheneos.net
mta-sts.grapheneos.network
mta-sts.grapheneos.online
mta-sts.grapheneos.org
mta-sts.grapheneos.ovh
mta-sts.grapheneos.page
mta-sts.grapheneos.social
mta-sts.mail.grapheneos.org
mta-sts.matrix.grapheneos.org
mta-sts.seamlessupdate.app
mta-sts.vanadium.app

- -

IPs:

- -

192.99.98.22 — OVH bhs6
2607:5300:205:200::472f — OVH bhs6

- -

Ports:

- -

TCP 22 ssh
TCP 25 smtp
TCP 80 http
TCP 443 https
TCP 465 submissions
TCP 993 imaps

- -

GrapheneOS discussion forum server

- -

Repository
Issue tracker

- -

Specs:

- -

OVH VPS vps2023-le-4
4 core
4 GB memory
80 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

discuss.grapheneos.org

- -

IPs:

- -

51.222.14.6 — OVH bhs6
2607:5300:205:200::29e8 — OVH bhs6

- -

Ports:

- -

TCP 22 ssh
TCP 80 http
TCP 443 https

- - - -

GrapheneOS Matrix server

- -

This server primarily runs the synapse Matrix server with PostgreSQL behind an - nginx web server. It also runs the mjolnir bot for moderation and matterbridge is - used to implement a bridge between Matrix, IRC and Telegram.

- -

Repository
Issue tracker

- -

Specs:

- -

OVH VPS vps2020-comfort-4-8-160
4 core
8 GB memory
160 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

matrix.grapheneos.org
element.grapheneos.org

- -

IPs:

- -

51.79.51.42 — OVH bhs6
2607:5300:205:200::26e1 — OVH bhs6

- -

Ports:

- -

TCP 22 ssh
TCP 80 http
TCP 443 https

- -

GrapheneOS DNS servers

- -

Repository
Issue tracker

- -

Specs:

- -

4x OVH VPS vps2023-le-2
2 core
2 GB memory
40 GB NVMe SSD storage
500 Mbit/s bandwidth

- -

3x BuyVM Slice 1024
1 core
1 GB memory
20 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

ns1.attestation.app
ns1.grapheneos.app
ns1.grapheneos.ca
ns1.grapheneos.com
ns1.grapheneos.dev
ns1.grapheneos.foundation
ns1.grapheneos.info
ns1.grapheneos.net
ns1.grapheneos.network
ns1.grapheneos.online
ns1.grapheneos.org
ns1.grapheneos.ovh
ns1.grapheneos.page
ns1.grapheneos.social
ns1.seamlessupdate.app
ns1.vanadium.app
ns2.attestation.app
ns2.grapheneos.app
ns2.grapheneos.ca
ns2.grapheneos.com
ns2.grapheneos.dev
ns2.grapheneos.foundation
ns2.grapheneos.info
ns2.grapheneos.net
ns2.grapheneos.network
ns2.grapheneos.online
ns2.grapheneos.org
ns2.grapheneos.ovh
ns2.grapheneos.page
ns2.grapheneos.social
ns2.seamlessupdate.app
ns2.vanadium.app

- -

IPs:

- -

185.187.152.9 (anycast), 51.161.34.158 (0.ns1.grapheneos.org) — OVH bhs6
2a05:b0c4:1::8 (anycast), 2607:5300:205:200::eaa (0.ns1.grapheneos.org) — OVH bhs6
185.187.152.9 (anycast), 15.204.8.153 (1.ns1.grapheneos.org) — OVH US us-west-or-2
2a05:b0c4:1::8 (anycast), 2604:2dc0:202:300::23a6 (1.ns1.grapheneos.org) — OVH us-west-or-2
185.187.152.9 (anycast) 57.129.65.223 (2.ns1.grapheneos.org) — OVH de2
2a05:b0c4:1::8 (anycast) 2001:41d0:701:1100::245b (2.ns1.grapheneos.org) — OVH de2
185.187.152.9 (anycast) 15.235.197.61 (3.ns1.grapheneos.org) — OVH sgp2
2a05:b0c4:1::8 (anycast) 2402:1f00:8000:800::3966 (3.ns1.grapheneos.org) — OVH sgp2
198.251.90.93 (anycast), 198.98.53.141 (0.ns2.grapheneos.org) — BuyVM New York
2605:6400:10:102e:95bc:89ef:2e7f:49bb (0.ns2.grapheneos.org) — BuyVM New York
198.251.90.93 (anycast), 205.185.124.155 (1.ns2.grapheneos.org) — BuyVM Las Vegas
2605:6400:20:1c8f:a0c9:372d:482e:945b (1.ns2.grapheneos.org) — BuyVM Las Vegas
198.251.90.93 (anycast), 107.189.3.168 (2.ns2.grapheneos.org) — BuyVM Luxembourg
2605:6400:30:ec25:102c:af6d:5be:1eb8 (2.ns2.grapheneos.org) — BuyVM Luxembourg

- -

Ports:

- -

TCP 22 ssh
TCP 53 domain
UDP 53 domain
TCP 80 http
TCP 443 https
TCP 853 domain-s

- -

Staging GrapheneOS DNS server

- -

Repository
Issue tracker

- -

Specs:

- -

BuyVM Slice 1024
1 core
1 GB memory
20 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

ns1.staging.attestation.app
ns2.staging.attestation.app
ns1.staging.grapheneos.org
ns2.staging.grapheneos.org

- -

IPs:

- -

198.98.56.238 — BuyVM New York
2605:6400:10:c41:de92:c534:326a:711a — BuyVM New York

- -

Ports:

- -

TCP 22 ssh
TCP 53 domain
UDP 53 domain
TCP 80 http
TCP 443 https
TCP 853 domain-s

- -

Attestation website and service

- -

Repository
Issue tracker

- -

Specs:

- -

OVH VPS vps2023-le-4
4 core
4 GB memory
80 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

attestation.app
www.attestation.app

- -

IPs:

- -

51.79.66.27 — OVH bhs6
2607:5300:205:200::7e9 — OVH bhs6

- -

Ports:

- -

TCP 22 ssh
TCP 80 http
TCP 443 https

- -

Staging attestation website and service

- -

Repository
Issue tracker

- -

Specs:

- -

BuyVM Slice 1024
1 core
1 GB memory
20 GB NVMe SSD storage
1000 Mbit/s bandwidth

- -

Domains:

- -

staging.attestation.app

- -

IPs:

- -

198.98.57.157 — BuyVM New York
2605:6400:10:aa9:1c0f:44d3:da15:c0ec — BuyVM New York

- -

Ports:

- -

TCP 22 ssh
TCP 80 http
TCP 443 https

Articles

- -

The main documentation for GrapheneOS is at the top-level of the site:

- -

Features overview — Overview of GrapheneOS features differentiating it from the Android Open Source Project.
Install — Installation instructions for GrapheneOS.
Build — Building instructions for GrapheneOS.
Usage guide — Usage instructions for GrapheneOS.
Frequently Asked Questions — Answers to frequently asked questions about GrapheneOS.
Releases changelog — Changelog for official releases of GrapheneOS.
Source code — Documenting all source code repositories for GrapheneOS.
History — History of the GrapheneOS project.

- -

Our attestation service has a page - explaining how the Auditor app and attestation service work.

- -

Other articles on assorted topics related to GrapheneOS:

- -

Attestation compatibility guide — Guide on using remote attestation in a way that's compatible with GrapheneOS
GrapheneOS servers — Documentation on GrapheneOS servers.
Server traffic shaping — Implementing server traffic shaping on Linux with CAKE.
Sitewide advertising industry opt-out — Using ads.txt / app-ads.txt to disallow buying or selling ads for a domain.

Positon location service

- -

The Positon location service is a proprietary and highly privacy invasive service - created by developers tied to /e/OS with their funding. There's a deliberate effort to - hide that it's tied to them in order to convince other projects to adopt it, as opposed - to using the similar service they host for /e/OS itself. Using the service requires - uploading sensitive location data to obtain location estimates, similar to the Apple and - Google location services. As with the Apple and Google services, it's a centralized - proprietary service with fully proprietary data. Unlike those services, the people - behind it have a history of publishing notoriously insecure software such as the /e/OS - operating system itself which massively rolls back standard security, lags years behind - on security updates and covers all of that up. They blatantly scam their users with - false privacy/security claims for /e/OS, and nothing different should be expected from a - location service from the same group of people. Multiple people involved in it are also - actively participating in harassment targeting privacy/security researchers and - engineers including but not limited to GrapheneOS team members.

- -

The people behind the Positon location service have repeatedly talked about the - importance they see in centralizing the whole open source community around using their - service while locking out alternatives to it through proprietary data. They have spread - fear, uncertainty and doubt about making services using open mapping data through - claiming that it's a privacy hazard for people to have access to maps of Wi-Fi networks - publicly broadcasting their SSID despite that data already being available through many - commercial providers including publicly queryable databases such as Wigle. Anyone can - drive around building these maps and many companies have already built them, with the - data available for sale, as Positon shows with them obtaining access to it. The real - privacy hazard is sending your location in real time to a service, particularly a poorly - secured one from people known to cover up and downplay vulnerabilities. Positon has been - built to grab as much market share as possible early on before actual open options can - emerge and gather the necessary data.

- -

The people involved in Positon have only ever cared about their careers, power and - influence. They've consistently been on a side against real privacy and security, but - rather focused on monetizing people's demand for it and grabbing as much market share as - they can as quickly as they can with endless false marketing and attacks on projects - like GrapheneOS. They see GrapheneOS as a huge threat to them due to us striving to - bring people real privacy and security at no cost, which is far easier to obtain and - use. This invalidates the business model of their companies like Murena. They - consistently use their non-profits mainly as a way to earn money and promote their - for-profit initiatives.

- -

The service claims to be free of charge, but a core goal is turning it into a way to - get data from users to build their own database that's largely not going to be available - for use by others. Using it is helping them build a future business at the expense of - user privacy, little different from the Apple and Google services. This is not what the - open source community needs from a location service. The claims of no strings attached - and the implication that it's open are nonsense. Storing as little data as possible - would mean using local database for the region, not a network-based service. They're - opposed to doing a local service well rather than it being their long term goal. They - explicitly aim to lock out other alternatives and deter local location detection via - Wi-Fi.

Server traffic shaping

- -

This article covers implementing server traffic shaping on Linux with CAKE. The aim - is to provide fair usage of bandwidth between clients and consistently low latency - for dedicated and virtual servers provided by companies like OVH and others.

- -

Traffic shaping is generally discussed in the context of a router shaping traffic - for a local network with assorted clients connected. It also has a lot to offer on a - server where you don't control the network. If you control your own infrastructure - from the server to the ISP, you probably want to do this on the routers instead.

- -

This article was motivated by the serious lack of up-to-date information on this - topic elsewhere. It's very easy to implement on modern Linux kernels and the results - are impressive from extremely simple test cases to heavily loaded servers.

- -

Problem

- -

A server will generally be provisioned with a specific amount of bandwidth - enforced by a router in close proximity. This router acts as the bottleneck and - ends up being in charge of most of the queuing and congestion decisions. Unless - that's under your control, the best you can hope for is that the router is - configured to use fq_codel as the queuing discipline (qdisc) to - provide fair queuing between streams and low latency by preventing a substantial - backlog of data.

- -

Unfortunately, the Linux kernel still defaults to pfifo_fast - instead of the much saner fq_codel algorithm. This is changed by a - configuration file shipped with systemd, so most distributions using - systemd as init end up with a sane default. Debian removes that configuration and - doesn't set a sane default itself, and is widely used. Many server providers like - OVH do not appear to consistently use modern queue disciplines like - fq_codel within their networks, particularly at artificial - bottlenecks implementing rate limiting based on product tiers.

- -

If the bottleneck doesn't use fair queuing, division of bandwidth across - streams is very arbitrary and latency suffers under congestion. These issues are - often referred to as bufferbloat, and fq_codel is quite good at - resolving it.

- -

The fq_codel algorithm is far from perfect. It has issues with - hash collisions and more importantly only does fair queuing between streams. - Buffer bloat also isn't the only relevant issue. Clients with multiple connections - receive more bandwidth and a client can open a large number of connections to - maximize their bandwidth usage at the expense of others. Fair queuing is important - beyond as a solution to bufferbloat and there's more to fair queuing than doing it - only based on streams.

- -

Traditionally, web browsers open a bunch of HTTP/1.1 connections to each server - which ends up giving them an unfair amount of bandwidth. HTTP/2 is much friendlier - since it uses a single connection to each server for the entire browser. Download - managers take this to the extreme and intentionally use many connections to bypass - server limits and game the division of resources between clients.

- -

Solution

- -

Linux 4.19 and later makes it easy to solve all of these problems. The CAKE - queuing discipline provides sophisticated fair queuing based on destination and - source addresses with finer-grained fairness for individual streams.

- -

Unfortunately, simply enabling it as your queuing discipline isn't enough - since it's highly unlikely that your server is the network bottleneck. You need to - configure it with a bandwidth limit based on the provisioned bandwidth to move the - bottleneck under your control where you can control how traffic is queued.

- -

Results

- -

We've used an 100mbit OVH server for as a test platform for a case where - clients can easily max out the server bandwidth on their own. As a very simple - example, consider 2 clients with more than 100mbit of bandwidth each downloading a - large file. These are (rounded) real world results with CAKE:

- -

client A with 1 connection gets 50mbit
client B with 10 connections gets 5mbit each adding up to 50mbit

- -

CAKE with flows instead of the default triple-isolate to - mimic fq_codel at a bottleneck:

- -

client A with 1 connection gets 9mbit
client B with 10 connections gets 9mbit each adding up to 90mbit

- -

The situation without traffic shaping is a mess. Latency takes a serious hit - that's very noticeable via SSH. Bandwidth is consistently allocated very unevenly - and ends up fluctuating substantially between test runs. The connections tend to - settle near a rate, often significantly lower or higher than the fair 9mbit - amount. It's generally something like this, but the range varies a lot:

- -

client A with 1 connection gets ~6mbit to ~14mbit
client B with 10 connections gets ~6mbit to ~14mbit each adding up to ~86mbit - to ~94mbit

- -

CAKE continues working as expected with a far higher number of connections. It - technically has a higher CPU cost than fq_codel, but that's much more - of a concern for low end router hardware. It hardly matters on a server, even one - that's under heavy CPU load. The improvement in user experience is substantial and - it's very noticeable in web page load speeds when a server is under load.

- -

Implementation

- -

For a server with 2000mbit of bandwidth provisioned, you could start by trying - it with 99.75% of the provisioned bandwidth:

- -

tc qdisc replace dev eth0 root cake bandwidth 1995mbit besteffort

- -

On a server, setting it to use 100% of the provisioned bandwidth may work fine - in practice. Unlike a local network connected to a consumer ISP, you shouldn't - need to sacrifice anywhere close to the typically recommended 5-10% of your - bandwidth for traffic shaping.

- -

This also sets besteffort for the common case where the server - doesn't have appropriate Quality of Service markings set up via Diffserv. Fair - scheduling is already great at providing low latency by cycling through the hosts - and streams without needing this kind of configuration. The defaults for Diffserv - traffic classes like real-time video are set up to yield substantial bandwidth in - exchange for lower latency. It's easy to set this up wrong and it usually won't - make much sense on a server. You might want to set up marking low priority traffic - like system updates, but it will already get a tiny share of the overall traffic - on a loaded server due to fair scheduling between hosts and streams.

- -

You can use the tc -s qdisc command to monitor CAKE:

- -

tc -s qdisc show dev eth0

- -

If you want to keep an eye on how it changes over time:

- -

watch -n 1 tc -s qdisc show dev eth0

- -

This is very helpful for figuring out if you've successfully moved the - bottleneck to the server. If the bandwidth is being fully used, it should - consistently have a backlog of data where it's applying the queuing discipline. - The backlog shouldn't be draining to near zero under full bandwidth usage as that - indicates the bottleneck is the server application itself or a different network - bottleneck.

- -

If you use systemd-network, you can add a CAKE configuration section to the - network configuration file instead of manually running the tc command - with a Type=oneshot service on boot:

- -

[CAKE]
-Bandwidth=1995M
-PriorityQueueingPreset=besteffort

- -

Quicker backpressure propagation

- -

The Linux kernel can be tuned to more quickly propagate TCP backpressure up to - applications while still maximizing bandwidth usage. This is incredibly useful for - interactive applications aiming to send the freshest possible copy of data and for - protocols like HTTP/2 multiplexing streams/messages with different priorities over - the same TCP connection. This can also substantially reduce memory usage for TCP - by reducing buffer sizes closer to the optimal amount for maximizing bandwidth - use without wasting memory. The downside to quicker backpressure propagation is - increased CPU usage from additional system calls and context switches.

- -

The Linux kernel automatically adjusts the size of the write queue to maximize - bandwidth usage. The write queue is divided into unacknowledged bytes (TCP window - size) and unsent bytes. As acknowledgements of transmitted data are received, it - frees up space for the application to queue more data. The queue of unsent bytes - provides the leeway needed to wake the application and obtain more data. This can - be reduced using net.ipv4.tcp_notsent_lowat to reduce the default and - the TCP_NOTSENT_LOWAT socket option to override it per-socket.

- -

A reasonable choice for internet-based workloads concerned about latency and - particularly prioritization within TCP connections but unwilling to sacrifice - throughput is 128kiB. To configure this, set the following in - /etc/sysctl.d/local.conf or another sysctl configuration file and - load it with sysctl --system:

- -

net.ipv4.tcp_notsent_lowat = 131072

- -

Using values as low as 16384 can make sense to further improve latency and - prioritization. However, it's more likely to negatively impact throughput and will - further increase CPU usage. Use at least 128k or the default of not limiting the - automatic unsent buffer size unless you're going to do substantial testing to make - sure there's not a negative impact for the workload.

- -

If you decide to use tcp_notsent_lowat, be aware that newer Linux - kernels (Linux 5.0+ with a further improvement for Linux 5.10+) are recommended to - substantially reduce system calls / context switches by not triggering the - application to provide more data until over half the unsent byte buffer is - empty.

- -

High link speed

- -

By default, CAKE splits General Segmentation Offload (GSO) super-packets to - reduce latency at the expense of CPU efficiency and throughput. This can create a - bottleneck at high link speeds. We've had to disable this on the 2Gbit GrapheneOS - update servers.

- -

[CAKE]
-Bandwidth=1995M
-PriorityQueueingPreset=besteffort
-SplitGSO=false

- -

Future

- -

Ideally, data centers would deploy CAKE throughout their networks with the - default triple-isolate flow isolation. This may mean they need to use - more powerful hardware for routing. If the natural bottlenecks used CAKE, setting - up traffic shaping on the server wouldn't be necessary. This doesn't seem likely - any time soon. Deploying fq_codel is much more realistic and tackles - buffer bloat but not the issue of fairness between hosts rather than only - streams.

Sitewide advertising industry opt-out

- -

The ads.txt specification - provides a way to list the authorized sellers of ads for a domain. The - app-ads.txt specification - extends this to cover apps tied to the domain. As a domain owner, this is a valuable - way to crack down on fraudulent usage of your domain including by adware.

- -

For domains without any third party advertising including those without any ads at - all, you should serve both /ads.txt and /app-ads.txt from a - web server with the placeholder record defined by the specification:

- -

placeholder.example.com, placeholder, DIRECT, placeholder

- -

The placeholder record formally disallows buying and selling ads on behalf of the - domain including for any subdomains. This prevents fraudulently buying / selling ads - for your domain anywhere that ads.txt / app-ads.txt are enforced.

- -

It's in the interest of most ad tech companies to enforce these standards due to - losses from ad fraud so adoption is increasingly widespread.

- -

Browser extension malware injecting ads into sites is very common and this is a way - for sites to hurt those malware developers where it hurts: their pocketbook.

- -

These standards have a limited scope and were primarily created to address the cost - of ad fraud for the advertising industry, but they do offer value for domain owners to - protect their reputation and discourage adware.