Index RSS

Enabling gzip compression

I have finally taken the time to enable gzip compression for this blog. OpenBSD's httpd(8) does support it via a simple option that you have to enable in your httpd.conf file (gzip-static), but it is not quite as simple as that.

With most http servers, you can just enable compression support and the corresponding content will be compressed on-the-fly if a client supports compression. This sounds like the best solution for everyone, but mind the details: Do we just compress it whenever a client asks for it? Do we cache the compressed files? What if gzip compression does not actually reduce the size, e.g. because the source file already is highly compressed data?

While not quite as comfortable, OpenBSD's httpd has a very simple answer for this: It does not ever compress anything by itself - you have to do it yourself! The "gzip-static" option works like this: If a client supports gzip compression and requests a resource, httpd will check whether a file with the same name and added suffix '.gz' exists. If it does, the contents of this file is sent back to the client, otherwise, it returns the plain file.

This scheme has the advantage that it is easy to control. I can always tell which files support compression and when the cpu load for compressing the files will occur. I can even distribute that cpu load to a different machine and just copy compressed files back into my /var/www/htdocs directory - that is kind of nifty!

The disadvantage of course is that I have to ensure myself that all files have corresponding compressed files and that the compressed versions are not stale copies of previous versions. And it takes more disk space, since I have to store both the compressed and the uncompressed versions of the files (assuming I want to support clients which do not support gzip compression - which I do!).

Automation of creating compressed files

Just enabling gzip-static is a one-liner in my configuration file and does not really make a good blog post. But I am a developer and as such am naturally lazy; I do not want to manage my compressed files myself. I am willing to sacrifice some of the control I have mentioned before for not having to worry about it (it is always good to have the choice either way).

I have a very simple deployment setup for this blog: After generating the html pages from my self-made template files, I copy them to a specific directory, which also is a git repository. There, I commit them and then just push to deploy. The remote end lies on my server in the /var/www/htdocs directory as a non-bare repository, which is automatically updated on receiving that push. In case you are wondering, this is made possible by configuring the remote end repository this way: 

git config receive.denyCurrentBranch updateInstead

And since I am using git this way, I can simply add a post-update hook to my git repository. I wrote this script and put it into the .git/hooks directory as 'post-update': 

#!/usr/bin/perl
use v5.40;
use warnings;

sub changed_files {
    my @lastrefs = `git reflog -2 --format='%h'`;
    chomp $lastrefs[0];

    if (defined $lastrefs[1]) {
	chomp $lastrefs[1];
	my $files = `git diff --name-only "$lastrefs[1]..$lastrefs[0]"`;
	chomp $files;
	return split /\n/, $files;
    } else {
	my $files = `git ls-files`;
	chomp $files;
	return split /\n/, $files;
    }
}

my @files = changed_files();

chdir `git rev-parse --show-toplevel`; # now we are in .git
chdir '..'; # now we are in the root directory

foreach my $file (@files) {
    say "not found: $file" unless -f $file;
    my $target = "$file.gz";
    unlink $target if -f $target;
    my $result = system("gzip -k -o '$target' '$file'");
    if ($result) {
	say "Error code $result while compressing $file";
    } else {
	say "$file -> $target";
    }
}

Don't mind the 'perl v5.40' requirement; it should run on most semi-current perl versions - if I write a perl script (and not a library), I like to set the version to the most current version available to all systems that I intend to run it on. For OpenBSD 7.8, that is perl v5.40 currently. If you wonder why perl: personal preferences aside, did you know that each git installation comes with a perl interpreter, even on windows? Anyway ...

I have made this script more generic than necessary, since it should also work if you are pushing into an empty repository, which I do not currently need. But I took the easy way for compressing the files by shelling out to an external gzip command.

With this, I will not have to think about compression for later edits, but I have to create the first compressed files by hand. I know loops in shell, so I am fine with that.

Does it work?

Let's check whether my blog now supports compression. Using 

curl -v --compressed -I "https://astharoshe.net/"

I get this response before enabling the compression option: 

curl -v --compressed -I "https://astharoshe.net/"

* Host astharoshe.net:443 was resolved.
* IPv6: 2a03:6000:6e65:629::39
* IPv4: 46.23.90.39
*   Trying [2a03:6000:6e65:629::39]:443...
* Immediate connect fail for 2a03:6000:6e65:629::39: No route to host
*   Trying 46.23.90.39:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL Trust Anchors:
*   CAfile: /etc/ssl/cert.pem
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*   subject: CN=astharoshe.net
*   start date: Dec 17 05:08:50 2025 GMT
*   expire date: Mar 17 05:08:49 2026 GMT
*   issuer: C=US; O=Let's Encrypt; CN=R13
*   Certificate level 0: Public key type ? (4096/128 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type ? (4096/128 Bits/secBits), signed using sha256WithRSAEncryption
*   subjectAltName: "astharoshe.net" matches cert's "astharoshe.net"
* SSL certificate verified via OpenSSL.
* Established connection to astharoshe.net (46.23.90.39 port 443) from 192.168.2.17 port 28591 
* using HTTP/1.x
> HEAD / HTTP/1.1
> Host: astharoshe.net
> User-Agent: curl/8.17.0
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
* Request completely sent off
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Cache-Control: public, no-cache, no-transform, must-revalidate, max-age=1209600
Cache-Control: public, no-cache, no-transform, must-revalidate, max-age=1209600
< Connection: keep-alive
Connection: keep-alive
< Connection: close
Connection: close
< Content-Length: 7460
Content-Length: 7460
< Content-Type: text/html
Content-Type: text/html
< Date: Wed, 31 Dec 2025 14:32:01 UTC
Date: Wed, 31 Dec 2025 14:32:01 UTC
< Last-Modified: Wed, 01 Oct 2025 11:26:48 UTC
Last-Modified: Wed, 01 Oct 2025 11:26:48 UTC
< Permissions-Policy: interest-cohort=()
Permissions-Policy: interest-cohort=()
< Server: OpenBSD httpd
Server: OpenBSD httpd
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< 

* shutting down connection #0

And this is the new response, after enabling compression: 

curl -v --compressed -I "https://astharoshe.net/"

* Host astharoshe.net:443 was resolved.
* IPv6: 2a03:6000:6e65:629::39
* IPv4: 46.23.90.39
*   Trying [2a03:6000:6e65:629::39]:443...
* Immediate connect fail for 2a03:6000:6e65:629::39: No route to host
*   Trying 46.23.90.39:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL Trust Anchors:
*   CAfile: /etc/ssl/cert.pem
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*   subject: CN=astharoshe.net
*   start date: Dec 17 05:08:50 2025 GMT
*   expire date: Mar 17 05:08:49 2026 GMT
*   issuer: C=US; O=Let's Encrypt; CN=R13
*   Certificate level 0: Public key type ? (4096/128 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type ? (4096/128 Bits/secBits), signed using sha256WithRSAEncryption
*   subjectAltName: "astharoshe.net" matches cert's "astharoshe.net"
* SSL certificate verified via OpenSSL.
* Established connection to astharoshe.net (46.23.90.39 port 443) from 192.168.2.17 port 19202 
* using HTTP/1.x
> HEAD / HTTP/1.1
> Host: astharoshe.net
> User-Agent: curl/8.17.0
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
* Request completely sent off
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Cache-Control: public, no-cache, no-transform, must-revalidate, max-age=1209600
Cache-Control: public, no-cache, no-transform, must-revalidate, max-age=1209600
< Connection: keep-alive
Connection: keep-alive
< Connection: close
Connection: close
< Content-Encoding: gzip
Content-Encoding: gzip
< Content-Length: 1585
Content-Length: 1585
< Content-Type: text/html
Content-Type: text/html
< Date: Wed, 31 Dec 2025 14:37:50 UTC
Date: Wed, 31 Dec 2025 14:37:50 UTC
< Last-Modified: Wed, 01 Oct 2025 11:26:48 UTC
Last-Modified: Wed, 01 Oct 2025 11:26:48 UTC
< Permissions-Policy: interest-cohort=()
Permissions-Policy: interest-cohort=()
< Server: OpenBSD httpd
Server: OpenBSD httpd
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< 

* shutting down connection #0

Looks good to me.

And my script? It makes nice little output lines like these during git push: 

remote: 2025-06-18-Rust_compile_time_reflections.html -> 2025-06-18-Rust_compile_time_reflections.html.gz
remote: 2025-12-31-Enabling_gzip_compression.html -> 2025-12-31-Enabling_gzip_compression.html.gz
remote: index.html -> index.html.gz
remote: rss.xml -> rss.xml.gz
remote: tags/openbsd.html -> tags/openbsd.html.gz

Why even compress http responses?

My blog mostly consists of small text files, that is, of files that can be compressed reasonably well, but that do not seem to really profit from compression. They were small to begin with, after all. Taking the example output from above, I went from 7460 bytes to 1585 bytes. What does it bring me?

Not much, I believe, to be honest. You would have to have a very small bandwidth to notice this. But the bigger my articles are, the more I feel that giving people the option to save some bandwidth is morally correct, and it might even be faster at some point.

At least it does not cost my much to enable this, and it was interesting to learn how to enable post-update git hooks for this use case.

Appendix: My httpd.conf



server "astharoshe.net" {
        alias "www.astharoshe.net"
        alias "files.astharoshe.net"

        listen on 127.0.0.1 port 80
        listen on ::1 port 80
        block return 301 "https://$SERVER_NAME$REQUEST_URI"

	log style forwarded
}

server "www.astharoshe.net" {
	listen on 127.0.0.1 port 8080
	listen on ::1 port 8080

	block return 301 "https://astharoshe.net$REQUEST_URI"
}

server "astharoshe.net" {
	listen on 127.0.0.1 port 8080
	listen on ::1 port 8080

	root "/htdocs/astharoshe.net"

	location "/.git/*" {
		block return 404
	}

	location "/.well-known/acme-challenge/*" {
		root "/acme"
		request strip 2
		directory no auto index
	}

	log style forwarded

	gzip-static
}

server "files.astharoshe.net" {
	listen on 127.0.0.1 port 8080
	listen on ::1 port 8080

	root "/htdocs/files.astharoshe.net"

	location "/.git/*" {
		block return 404
	}

	location "/.well-known/acme-challenge/*" {
		root "/acme"
		request strip 2
		directory no auto index
	}

	location "/*" {
		directory auto index
	}

	log style forwarded
}

types {
	include "/usr/share/misc/mime.types"
}