From b18f8a0970fce67019033657f2a23498e0225a3a Mon Sep 17 00:00:00 2001 From: Dante Catalfamo Date: Thu, 18 Jun 2020 01:28:57 -0400 Subject: Rename posts content directory to post --- config.toml | 2 +- content/post/how-this-blog-works/index.org | 229 +++++++++++++++++++++ content/post/letsencrypt-on-openbsd/index.org | 122 +++++++++++ .../letsencrypt-on-openbsd/openbsd letsencrypt.png | Bin 0 -> 62340 bytes content/post/openvpn-issues-openbsd-6.7/index.org | 72 +++++++ .../openbsd protonvpn no connection.png | Bin 0 -> 73572 bytes content/post/pcengines-comparison/index.org | 96 +++++++++ .../post/pcengines-comparison/pc engines vs.png | Bin 0 -> 277966 bytes content/posts/how-this-blog-works/index.org | 229 --------------------- content/posts/letsencrypt-on-openbsd/index.org | 122 ----------- .../letsencrypt-on-openbsd/openbsd letsencrypt.png | Bin 62340 -> 0 bytes content/posts/openvpn-issues-openbsd-6.7/index.org | 72 ------- .../openbsd protonvpn no connection.png | Bin 73572 -> 0 bytes content/posts/pcengines-comparison/index.org | 96 --------- .../posts/pcengines-comparison/pc engines vs.png | Bin 277966 -> 0 bytes 15 files changed, 520 insertions(+), 520 deletions(-) create mode 100644 content/post/how-this-blog-works/index.org create mode 100644 content/post/letsencrypt-on-openbsd/index.org create mode 100644 content/post/letsencrypt-on-openbsd/openbsd letsencrypt.png create mode 100644 content/post/openvpn-issues-openbsd-6.7/index.org create mode 100644 content/post/openvpn-issues-openbsd-6.7/openbsd protonvpn no connection.png create mode 100644 content/post/pcengines-comparison/index.org create mode 100644 content/post/pcengines-comparison/pc engines vs.png delete mode 100644 content/posts/how-this-blog-works/index.org delete mode 100644 content/posts/letsencrypt-on-openbsd/index.org delete mode 100644 content/posts/letsencrypt-on-openbsd/openbsd letsencrypt.png delete mode 100644 content/posts/openvpn-issues-openbsd-6.7/index.org delete mode 100644 content/posts/openvpn-issues-openbsd-6.7/openbsd protonvpn no connection.png delete mode 100644 content/posts/pcengines-comparison/index.org delete mode 100644 content/posts/pcengines-comparison/pc engines vs.png diff --git a/config.toml b/config.toml index c8503fd..63c06f1 100644 --- a/config.toml +++ b/config.toml @@ -24,7 +24,7 @@ enableRobotsTXT = true # images = ["img/og.png"] # This will use the image called og.png in static/img folder # Posts shown in homepage - mainSections = ["posts"] + mainSections = ["post"] [menu] [[menu.nav]] diff --git a/content/post/how-this-blog-works/index.org b/content/post/how-this-blog-works/index.org new file mode 100644 index 0000000..d48dd37 --- /dev/null +++ b/content/post/how-this-blog-works/index.org @@ -0,0 +1,229 @@ +#+TITLE: Creating Self-Hosted Hugo Blog with OpenBSD +#+DATE: 2020-06-17T21:03:26-04:00 +#+DRAFT: true +#+DESCRIPTION: +#+TAGS[]: hugo openbsd +#+KEYWORDS[]: +#+SLUG: +#+SUMMARY: + +When creating this blog, there were a couple of factors I kept in mind +while trying to figure out how I was going to set it up. Here's an +approximate list: + +- Simple +- Version controlled +- Easy to host on OpenBSD +- Minimal maintenance +- Good integration with Emacs + +That's how I came up with what I currently use. Let me walk you +through how I run by blog. + +* Framework + + The blog engine is [[https://gohugo.io/][hugo]], a static site generator. I chose this over + something dynamic like wordpress for several reasons. + + First of all, it's very easy to manage, blog posts are just simple + files written in one of the markup languages hugo supports. Being + statically generated is also a massive advantage in terms of + maintenance. With something like wordpress, you have to be careful to + keep your site and plugins up to date. + + Since there's no dynamic content generation or database involved with + hugo, the attack surface is dramatically decreased. No possibility for + SQL injection, PHP RATs, accidental shell access, or hacked + credentials. The entire site is generated using a single command after + a new post is created, and then moved to the web server's root + directory. + + Being all flat files also means the entire thing can very easily be + tracked using =git= (or maybe [[https://gameoftrees.org/][got]], eventually), in fact that's the + recommended way to use hugo. There's no fear I'll accidentally delete + something, as I can always go back to a previous commit and restore + it. + + Since hugo is written in go, it's trivial to compile on OpenBSD, and + is actually available directly from the OpenBSD package manager right + out of the gate. + + Maybe the most important thing to be however, is that hugo natively + supports org-mode markup. I write all my notes, both personal and work + related, in org-mode. It makes converting my notes into blog posts + really easy. It also lets me leverage my existing Emacs setup, which + comes in handy often. While it's not very well documented, since + org-mode markup is a bit of a second class citizen in the hugo world, + it's pretty easy to figure out. + +* Prerequisites + + This can be hosted on a very cheap VPS since it only has to serve + static pages. For OpenBSD hosting I would recommend either [[https://www.vultr.com/][Vultr]] or + [[https://openbsd.amsterdam/][OpenBSD Amsterdam]]. + +** Server + The only thing that's required on the host server is =git=, although + you could even get away without that if you chose to host your git + repository elsewhere, like on github. + + #+BEGIN_SRC shell + pkg_add git + #+END_SRC + +** Local machine + On the local machine you'll need both =git= and =rsync=. Both might + already be installed depending on the system you're on. If not, + check your package manager for details on how to install them. In + the case of Ubuntu or Debian it would be + + #+BEGIN_SRC shell + sudo apt install git rsync + #+END_SRC + + or for Fedora + + #+BEGIN_SRC shell + sudo dnf install git rsync + #+END_SRC + +* Version Control + + I wanted to try to keep things as simple as possible for this. The + "origin" for the blog is simply a bare git repository in the =blog= + user's home directory. + +** Setting up the blog user + + First I set up the blog user + #+BEGIN_SRC shell + useradd -m blog + #+END_SRC + + I then placed my public SSH key in its =authorized_keys= file + + #+BEGIN_SRC shell + mkdir -m 700 /home/blog/.ssh + cp /root/.ssh/authorized_keys /home/blog/.ssh/ + chown -R blog:blog /home/blog + #+END_SRC + + I then logged in as the blog user and initialize the bare git + repository. + + #+BEGIN_SRC shell + su blog + cd # cd with no arguments goes to home directory + git init --bare blog.git + #+END_SRC + +** Cloning the repository + + Cloning the repository onto my local machine is very easy at this + point. As long as my private keys are in the =blog= user's + =authorized_keys=, git will take care of the rest. + + #+BEGIN_SRC shell + # on my local machine + git clone blog@lambda.cx:blog.git + #+END_SRC + + I can now work on the blog as I would any other git repository, + pulling with =git pull= and pushing with =git push=. + +* Web server + + Since this blog is going to be hosted on OpenBSD, we don't need to + install a web server, as it already comes with one built in. + + Setting up =httpd= couldn't be easier, the configuration syntax is + very straight forward. If you would like to see a full breakdown of + the options available, check out the man page with =man + httpd.conf=. The example configuration in =/etc/examples/httpd.conf= + is also a good starting point. + + In this case the simplest configuration would be as follows + + # js chosen for prism.js syntax highlighting + #+BEGIN_SRC js + server "blog.lambda.cx" { + listen on * port 80 + root "/htdocs/blog.lambda.cx" + } + #+END_SRC + + Despite how it looks, the =htdocs= folder doesn't reside in the + system root (=/=) directory. It actually lives in =/var/www/htdocs=, + and only appears that way because =httpd= gets automatically + =chroot='ed in =/var/www/= for security reasons. + + For more information about how to set up SSL with Let's Encrypt, + check out [[{{< ref "post/letsencrypt-on-openbsd" >}}][this post]]. + +* Using Hugo + + There's not much to say here. Hugo's [[https://gohugo.io/][website]] has good documentation + on how to get started creating a blog using their program. I'll be + covering the intricacies of using org-mode with hugo in the future. + +* Deployment + + The system used to deploy this blog is incredibly simple, involving + only =rsync=, =hugo=, and a small shell script. + +** Server + First you have to allow the =blog= user to write to the website + root. We'll do this by making it the owner of + =/var/www/htdocs/blog.lambda.cx=. + + #+BEGIN_SRC shell + chown -R blog /var/www/htdocs/blog.lambda.cx + #+END_SRC + +** Local machine + + This is the script used to deploy the website. It's placed in the + root of the hugo git repository. + + #+BEGIN_SRC shell + #!/bin/sh + set -e + + cd '$(dirname "$0")' + hugo + rsync -va --progress --rsync-path="/usr/bin/openrsync" public/ blog@lambda.cx:/var/www/htdocs/lambda.cx/blog + #+END_SRC + + Going through it line by line: + + - ~set -e~ Tells the shell to exit if any commands fail instead of + continuing execution. If ~hugo~ has a problem and exits with an + error, don't sync with the server. + - ~cd '$(dirname "$0")'~ Changes to the script's directory. This is + used in case you're running it from somewhere else. + - ~hugo~ Compile the website into static files located in the + =public= directory. + - ~rsync -va --progress --rsync-path="/usr/bin/openrsync" public/ + blog@lambda.cx:/var/www/htdocs/lambda.cx/blog~ This one is bigger + so I'll break it down. + - =rsync= A command that synchronizes files between two directories + - =-v= Be verbose, this is optional but I like it + - =-a= Stands for "archive": copy recursively, keep + permissions, etc. See the =rsync= man page if you're curious. + - =--progress= Show progress, also optional + - ~--rsync-path="/usr/bin/openrsync"~ This line is very important + for OpenBSD servers. OpenBSD has its own =rsync= implementation + called =openrsync=. Without this argument, =rsync= will connect + to the server, see that the =rsync= command doesn't exist, and + fail. + - =public/= Specify which folder we want to sync. The trailing + =/= is important. Without it =rsync= will copy the folder + instead of the folder's contents, which is what we want. + - =blog@lambda.cx:/var/www/htdocs/lambda.cx/blog= Login to user + =blog= on server =lambda.cx=, syncing files with the + =/var/www/htdocs/lambda.cx/blog= directory. + + The reason to use =rsync= here instead of something like =scp= is + that =rsync= won't upload files it doesn't need to, so if 3/4 of + the files didn't change when you updated the blog, it won't waste + time re-uploading them. diff --git a/content/post/letsencrypt-on-openbsd/index.org b/content/post/letsencrypt-on-openbsd/index.org new file mode 100644 index 0000000..556404b --- /dev/null +++ b/content/post/letsencrypt-on-openbsd/index.org @@ -0,0 +1,122 @@ +#+TITLE: Let's Encrypt on OpenBSD +#+DATE: 2020-06-16T22:56:27-04:00 +#+DRAFT: false +#+DESCRIPTION: Setting up acme-client on OpenBSD +#+TAGS[]: openbsd httpd letsencrypt acme-client +#+KEYWORDS[]: openbsd httpd letsencrypt acme-client +#+SLUG: +#+SUMMARY: + +#+ATTR_HTML: :alt Let's Encrypt OpenBSD +#+ATTR_HTML: :title Let's Encrypt OpenBSD +[[file:openbsd%20letsencrypt.png]] + +So I have an OpenBSD server serving a static website using +=httpd=. I've been thinking for a while I should add an SSL +certificate, but never got around to it because it was just a small +hobby website and it didn't require any real attention. + +Today while watching one of the OpenBSD tutorials at BSDCan, I thought +it was finally time. Since configuring everything else in OpenBSD is +so easy, this must be easy too, right? + +These were the only changes I had to make to my =httpd.conf= to get +=acme-client= to work. This is described in the =acme-client= man +page. +#+BEGIN_SRC diff +--- httpd.conf ++++ httpd.conf.new +@@ -1,4 +1,19 @@ + server "lambda.cx" { + listen on * port 80 + root "/htdocs/lambda.cx" ++ location "/.well-known/acme-challenge/*" { ++ root "/acme" ++ request strip 2 ++ } + } +#+END_SRC + +After that, I reloaded =httpd= with ~rcctl reload httpd~ + +I then copied the example config from =/etc/examples/acme-client.conf= +to =/etc/acme-client=. This is what the modifications to the example I +made look like. + +#+BEGIN_SRC diff +--- acme-client.conf ++++ acme-client.conf.new +@@ -1,19 +1,19 @@ + # + # $OpenBSD: acme-client.conf,v 1.2 2019/06/07 08:08:30 florian Exp $ + # + authority letsencrypt { + api url "https://acme-v02.api.letsencrypt.org/directory" + account key "/etc/acme/letsencrypt-privkey.pem" + } + + authority letsencrypt-staging { + api url "https://acme-staging-v02.api.letsencrypt.org/directory" + account key "/etc/acme/letsencrypt-staging-privkey.pem" + } + +-domain example.com { +- alternative names { secure.example.com } +- domain key "/etc/ssl/private/example.com.key" +- domain full chain certificate "/etc/ssl/example.com.fullchain.pem" ++domain lambda.cx { ++ # alternative names { www.lambda.cx } ++ domain key "/etc/ssl/private/lambda.cx.key" ++ domain full chain certificate "/etc/ssl/lambda.cx.fullchain.pem" + sign with letsencrypt + } +#+END_SRC + +It's a pretty small change. I have the alternative name line commented +out because I only have =lambda.cx= pointing at my server and not +=www.lambda.cx=. Although if I did I would un-comment it. I could also +add sub-domains like =sub.lambda.cx= in that area separated by a +space. + +After that I just had to run ~acme-client -v lambda.cx~ (-v for +verbosity) and it generated the certificates. + +Then I added a =crontab= entry (using =crontab -e=) to run once a day +at a random time and reload =httpd=. + +#+BEGIN_SRC +~ ~ * * * acme-client lambda.cx && rcctl reload httpd +#+END_SRC + +Finally to use the new certificates I added the following lines to my +=httpd.conf=. + +#+BEGIN_SRC diff +--- httpd.conf ++++ httpd.conf.new +@@ -1,8 +1,21 @@ + server "lambda.cx" { + listen on * port 80 + root "/htdocs/lambda.cx" + location "/.well-known/acme-challenge/*" { + root "/acme" + request strip 2 + } + } ++ ++server "lambda.cx" { ++ listen on * tls port 443 ++ tls { ++ certificate "/etc/ssl/lambda.cx.fullchain.pem" ++ key "/etc/ssl/private/lambda.cx.key" ++ } ++ root "/htdocs/lambda.cx" ++ location "/.well-known/acme-challenge/*" { ++ root "/acme" ++ request strip 2 ++ } ++} +#+END_SRC + +I reloaded httpd with ~rcctl reload httpd~ and that was it, working +certificate! diff --git a/content/post/letsencrypt-on-openbsd/openbsd letsencrypt.png b/content/post/letsencrypt-on-openbsd/openbsd letsencrypt.png new file mode 100644 index 0000000..f805be0 Binary files /dev/null and b/content/post/letsencrypt-on-openbsd/openbsd letsencrypt.png differ diff --git a/content/post/openvpn-issues-openbsd-6.7/index.org b/content/post/openvpn-issues-openbsd-6.7/index.org new file mode 100644 index 0000000..a9c34bd --- /dev/null +++ b/content/post/openvpn-issues-openbsd-6.7/index.org @@ -0,0 +1,72 @@ +#+TITLE: Issues with OpenVPN on OpenBSD 6.7 +#+DATE: 2020-06-14T14:08:06-04:00 +#+DRAFT: false +#+DESCRIPTION: +#+TAGS[]: openvpn openbsd libressl +#+KEYWORDS[]: +#+SLUG: +#+SUMMARY: + +#+ATTR_HTML: :alt No connection to ProtonVPN from OpenBSD +#+ATTR_HTML: :title No connection to ProtonVPN from OpenBSD +[[file:openbsd%20protonvpn%20no%20connection.png]] + +I have an OpenBSD VPN gateway I use to send all traffic it receives +over a VPN connection, and I noticed that no traffic was going through. + +I'd been using ProtonVPN as my provider for months prior to this +without any issues, so it was very confusing when it stopped working. + +No matter which VPN profile I used from ProtonVPN, it always gets +stuck after the step =TLS: Initial packet from +[AF_INET]XXX.XXX.XXX.XXX:YY=. Regardless of whether I tried the +individual server profiles, country profiles, free, or plus profiles. + +I tried starting openvpn with maximum verbosity. Everything launched +exactly as it should, until it gets to the TLS handshake, where it +failed to get a response. + +#+BEGIN_SRC +Sun Jun 14 15:37:22 2020 us=577519 UDP link local: (not bound) +Sun Jun 14 15:37:22 2020 us=577532 UDP link remote: [AF_INET]XXX.XXX.XXX.XXX:YYYY +Sun Jun 14 15:37:22 2020 us=577650 UDP WRITE [86] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 pid=[ #1 ] [ ] pid=0 DATA len=0 +Sun Jun 14 15:37:22 2020 us=739355 UDP READ [98] from [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_HARD_RESET_SERVER_V2 kid=0 pid=[ #1 ] [ 0 ] pid=0 DATA len=0 +Sun Jun 14 15:37:22 2020 us=739517 TLS: Initial packet from [AF_INET]XXX.XXX.XXX.XXX:YYYY, sid=19fe5aac 2d00f4aa +Sun Jun 14 15:37:22 2020 us=739658 UDP WRITE [94] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_ACK_V1 kid=0 pid=[ #2 ] [ 0 ] +Sun Jun 14 15:37:22 2020 us=739798 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this +Sun Jun 14 15:37:22 2020 us=739900 UDP WRITE [331] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_V1 kid=0 pid=[ #3 ] [ ] pid=1 DATA len=245 +Sun Jun 14 15:37:24 2020 us=832019 UDP WRITE [331] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_V1 kid=0 pid=[ #4 ] [ ] pid=1 DATA len=245 +Sun Jun 14 15:37:29 2020 us=32189 UDP WRITE [331] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_V1 kid=0 pid=[ #5 ] [ ] pid=1 DATA len=245 +#+END_SRC + +It just kept repeating the write until it timed out after 60 +seconds. It was like this for every country, on every port. Even using +the TCP profiles, but instead there the connection would get reset +almost immediately instead of timing out. + +I tried several free VPNs I found online just to compare, and all of +them worked without issue. This problem has only happened for me with +ProtonVPN servers. + +I tried connecting using both my desktop machine, and an Ubuntu VM, +both of which were able to connect without issue. The problem wasn't +with the account itself. + +I tried using another OpenBSD VM on my network, and the result was the +same as the VPN gateway, a timeout. I even spun up a fresh OpenBSD VM +in Vultr to see if the issue persisted on a new install in a different +network. The issue was still there. + +I was sure to check that the system clocks were correct on each +machine, and also tried commenting out all lines in the VPN profile +that weren't strictly required to make the connection, like mtu and +compression settings. + +As a final attempt, I tried installing OpenVPN with =mbedtls=. For all +my previous experiments, I had been using the default openvpn package, +which uses OpenBSD's LibreSSL. That time it worked perfectly. + +It occurred to me that this had been the first time I'd checked up on +the VPN gateway since updating to OpenBSD 6.7. I guess something about +a recent LibreSSL update has broken a feature OpenVPN relies on in +certain situations. diff --git a/content/post/openvpn-issues-openbsd-6.7/openbsd protonvpn no connection.png b/content/post/openvpn-issues-openbsd-6.7/openbsd protonvpn no connection.png new file mode 100644 index 0000000..f4fbe9b Binary files /dev/null and b/content/post/openvpn-issues-openbsd-6.7/openbsd protonvpn no connection.png differ diff --git a/content/post/pcengines-comparison/index.org b/content/post/pcengines-comparison/index.org new file mode 100644 index 0000000..a80f773 --- /dev/null +++ b/content/post/pcengines-comparison/index.org @@ -0,0 +1,96 @@ +#+TITLE: PC Engines APU Comparison +#+DATE: 2020-06-17T00:50:06-04:00 +#+DRAFT: false +#+DESCRIPTION: Comparison between PC Engines APU machines +#+TAGS[]: hardware pcengines +#+KEYWORDS[]: hardware pcengines +#+SLUG: +#+SUMMARY: + +#+ATTR_HTML: :alt PC Engines Comparison +#+ATTR_HTML: :title PC Engines Comparison +[[file:pc%20engines%20vs.png]] + +I've been looking at the [[https://www.pcengines.ch/apu2.htm][PC Engines APU]] line for a while. They're a +line of medium size single board PCs with a DB9 serial connector and +no VGA port. They also have gigabit Ethernet. Because of this they're +often used as firewall machines. + +I want to get one and use it as either the home router, or an +experimental server to mess around with. Quite a few OpenBSD folks use +them and recommend them as OpenBSD router and server hardware. They +aren't too expensive, have decent specs and a small physical +footprint. + +One thing that's always confused me was the naming scheme, which is a +little confusing at first. Initially there was the APU, then the APU2, +as described on their site, which makes sense. Then the APU model +numbers get a little confusing. They come in several variants, I'll +list them here for context. + +#+CAPTION: Taken from the PC Engines website +#+BEGIN_SRC + apu2d0 (2 GB DRAM, 2 i211AT NICs) + apu2e2 (2 GB DRAM, 3 i211AT NICs) + apu2e4 (4 GB DRAM, 3 i210AT NICs) + apu3c2 (2 GB DRAM, 3 i211AT NICs, optimized for 3G/LTE modems) + apu3c4 (4 GB DRAM, 3 i211AT NICs, optimized for 3G/LTE modems) + apu4d2 (2 GB DRAM, 4 i211AT NICs) + apu4d4 (4 GB DRAM, 4 i211AT NICs) + #+END_SRC + +What do the letters between the numbers mean? What is the significance +of the numbers in the first place? + +Let's take the =apu3c2= as an example. The =3= here means it's version +3 of the APU board. The APU 1 is no longer sold, so it's left out of +the list on the PC Engines website, along with most of their +re-sellers. The letter =c= is the revision of that board. It's pretty +much irrelevant for comparison. You'll want to get the most recent +one, as listed on the PC Engines website. The revisions are mostly +minor tweaks to the board. The final number, =2=, is the number of gigabytes +of RAM (in most cases). + +The APU2E2 boards have a single mSATA slot, and a regular SATA +connector, two mPCIe slots, and a SIM tray. It comes with 2 external +USB 3.0 type A ports, 2 internal USB 2.0 ports (header only), an SD +card slot, and a GPIO header. It also has 3 Intel i211AT gigabit +NICs. Using the SIM tray will remove the ability to use one of the two +mPCIe slots. + +So what are the major differences between version numbers? They all +use the same CPU, the quad core AMD Embedded G series GX-412TC running +at 1 GHz. Most other components on the boards are the same as +well. I'll give a description of the differences relative to the APU2 +board. + +The APU2D0 is the same as the regular APU2, but without a GPIO +headers, and it has only 2 NICs. + +The APU3 has an extra SIM tray, and the first slot, which is mSATA +only on the APU2, can be used as either an mSATA or USB 3G/LTE Modem +slot. The second slot can only be used for a modem. It also lets you +do GPIO tray swapping and failover, so you can use only a single modem +for 2 SIMs. + +The APU4D2 has 4 NICs instead of 3, and the second slot is modem +only. It has 2 GB of RAM. + +The APU4D4 is the same as the APU4D2, except it has dual SIM trays. It +has 4 GB of RAM. + +I found a lot of this information wasn't really obvious to upon first +reading the PC Engines website, as there's no direct comparison +between the board, and the site relies on you flipping between pages +to find the details. The re-seller website I looked at also didn't +elaborate on the differences. + +| Board | RAM | Slot 1 | Slot 2 | Slot 3 | Ethernet Controllers | SIM Slots | USB 2.0 | Headers | +|--------+------+-------------+-------------+--------+----------------------+-----------+---------+---------| +| [[https://www.pcengines.ch/apu2d0.htm][apu2d0]] | 2 GB | mSATA | mPCIe/Modem | mPCIe | 2 i211AT | 1 | 2 | No | +| [[https://www.pcengines.ch/apu2e2.htm][apu2e2]] | 2 GB | mSATA | mPCIe/Modem | mPCIe | 3 i211AT | 1 | 2 | Yes | +| [[https://www.pcengines.ch/apu2e4.htm][apu2e4]] | 4 GB | mSATA | mPCIe/Modem | mPCIe | 3 i210AT | 1 | 2 | Yes | +| [[https://www.pcengines.ch/apu3c2.htm][apu3c2]] | 2 GB | mSATA/Modem | Modem | mPCIe | 3 i211AT | 2 | 4 | Yes | +| [[https://www.pcengines.ch/apu3c4.htm][apu3c4]] | 4 GB | mSATA/Modem | Modem | mPCIe | 3 i211AT | 2 | 4 | Yes | +| [[https://www.pcengines.ch/apu4d2.htm][apu4d2]] | 2 GB | mSATA | Modem | mPCIe | 4 i211AT | 1 | 2 | Yes | +| [[https://www.pcengines.ch/apu4d4.htm][apu4d4]] | 4 GB | mSATA/Modem | Modem | mPCIe | 4 i211AT | 2 | 2 | Yes | diff --git a/content/post/pcengines-comparison/pc engines vs.png b/content/post/pcengines-comparison/pc engines vs.png new file mode 100644 index 0000000..88c715a Binary files /dev/null and b/content/post/pcengines-comparison/pc engines vs.png differ diff --git a/content/posts/how-this-blog-works/index.org b/content/posts/how-this-blog-works/index.org deleted file mode 100644 index da9c15d..0000000 --- a/content/posts/how-this-blog-works/index.org +++ /dev/null @@ -1,229 +0,0 @@ -#+TITLE: Creating Self-Hosted Hugo Blog with OpenBSD -#+DATE: 2020-06-17T21:03:26-04:00 -#+DRAFT: true -#+DESCRIPTION: -#+TAGS[]: hugo openbsd -#+KEYWORDS[]: -#+SLUG: -#+SUMMARY: - -When creating this blog, there were a couple of factors I kept in mind -while trying to figure out how I was going to set it up. Here's an -approximate list: - -- Simple -- Version controlled -- Easy to host on OpenBSD -- Minimal maintenance -- Good integration with Emacs - -That's how I came up with what I currently use. Let me walk you -through how I run by blog. - -* Framework - - The blog engine is [[https://gohugo.io/][hugo]], a static site generator. I chose this over - something dynamic like wordpress for several reasons. - - First of all, it's very easy to manage, blog posts are just simple - files written in one of the markup languages hugo supports. Being - statically generated is also a massive advantage in terms of - maintenance. With something like wordpress, you have to be careful to - keep your site and plugins up to date. - - Since there's no dynamic content generation or database involved with - hugo, the attack surface is dramatically decreased. No possibility for - SQL injection, PHP RATs, accidental shell access, or hacked - credentials. The entire site is generated using a single command after - a new post is created, and then moved to the web server's root - directory. - - Being all flat files also means the entire thing can very easily be - tracked using =git= (or maybe [[https://gameoftrees.org/][got]], eventually), in fact that's the - recommended way to use hugo. There's no fear I'll accidentally delete - something, as I can always go back to a previous commit and restore - it. - - Since hugo is written in go, it's trivial to compile on OpenBSD, and - is actually available directly from the OpenBSD package manager right - out of the gate. - - Maybe the most important thing to be however, is that hugo natively - supports org-mode markup. I write all my notes, both personal and work - related, in org-mode. It makes converting my notes into blog posts - really easy. It also lets me leverage my existing Emacs setup, which - comes in handy often. While it's not very well documented, since - org-mode markup is a bit of a second class citizen in the hugo world, - it's pretty easy to figure out. - -* Prerequisites - - This can be hosted on a very cheap VPS since it only has to serve - static pages. For OpenBSD hosting I would recommend either [[https://www.vultr.com/][Vultr]] or - [[https://openbsd.amsterdam/][OpenBSD Amsterdam]]. - -** Server - The only thing that's required on the host server is =git=, although - you could even get away without that if you chose to host your git - repository elsewhere, like on github. - - #+BEGIN_SRC shell - pkg_add git - #+END_SRC - -** Local machine - On the local machine you'll need both =git= and =rsync=. Both might - already be installed depending on the system you're on. If not, - check your package manager for details on how to install them. In - the case of Ubuntu or Debian it would be - - #+BEGIN_SRC shell - sudo apt install git rsync - #+END_SRC - - or for Fedora - - #+BEGIN_SRC shell - sudo dnf install git rsync - #+END_SRC - -* Version Control - - I wanted to try to keep things as simple as possible for this. The - "origin" for the blog is simply a bare git repository in the =blog= - user's home directory. - -** Setting up the blog user - - First I set up the blog user - #+BEGIN_SRC shell - useradd -m blog - #+END_SRC - - I then placed my public SSH key in its =authorized_keys= file - - #+BEGIN_SRC shell - mkdir -m 700 /home/blog/.ssh - cp /root/.ssh/authorized_keys /home/blog/.ssh/ - chown -R blog:blog /home/blog - #+END_SRC - - I then logged in as the blog user and initialize the bare git - repository. - - #+BEGIN_SRC shell - su blog - cd # cd with no arguments goes to home directory - git init --bare blog.git - #+END_SRC - -** Cloning the repository - - Cloning the repository onto my local machine is very easy at this - point. As long as my private keys are in the =blog= user's - =authorized_keys=, git will take care of the rest. - - #+BEGIN_SRC shell - # on my local machine - git clone blog@lambda.cx:blog.git - #+END_SRC - - I can now work on the blog as I would any other git repository, - pulling with =git pull= and pushing with =git push=. - -* Web server - - Since this blog is going to be hosted on OpenBSD, we don't need to - install a web server, as it already comes with one built in. - - Setting up =httpd= couldn't be easier, the configuration syntax is - very straight forward. If you would like to see a full breakdown of - the options available, check out the man page with =man - httpd.conf=. The example configuration in =/etc/examples/httpd.conf= - is also a good starting point. - - In this case the simplest configuration would be as follows - - # js chosen for prism.js syntax highlighting - #+BEGIN_SRC js - server "blog.lambda.cx" { - listen on * port 80 - root "/htdocs/blog.lambda.cx" - } - #+END_SRC - - Despite how it looks, the =htdocs= folder doesn't reside in the - system root (=/=) directory. It actually lives in =/var/www/htdocs=, - and only appears that way because =httpd= gets automatically - =chroot='ed in =/var/www/= for security reasons. - - For more information about how to set up SSL with Let's Encrypt, - check out [[{{< ref "posts/letsencrypt-on-openbsd" >}}][this post]]. - -* Using Hugo - - There's not much to say here. Hugo's [[https://gohugo.io/][website]] has good documentation - on how to get started creating a blog using their program. I'll be - covering the intricacies of using org-mode with hugo in the future. - -* Deployment - - The system used to deploy this blog is incredibly simple, involving - only =rsync=, =hugo=, and a small shell script. - -** Server - First you have to allow the =blog= user to write to the website - root. We'll do this by making it the owner of - =/var/www/htdocs/blog.lambda.cx=. - - #+BEGIN_SRC shell - chown -R blog /var/www/htdocs/blog.lambda.cx - #+END_SRC - -** Local machine - - This is the script used to deploy the website. It's placed in the - root of the hugo git repository. - - #+BEGIN_SRC shell - #!/bin/sh - set -e - - cd '$(dirname "$0")' - hugo - rsync -va --progress --rsync-path="/usr/bin/openrsync" public/ blog@lambda.cx:/var/www/htdocs/lambda.cx/blog - #+END_SRC - - Going through it line by line: - - - ~set -e~ Tells the shell to exit if any commands fail instead of - continuing execution. If ~hugo~ has a problem and exits with an - error, don't sync with the server. - - ~cd '$(dirname "$0")'~ Changes to the script's directory. This is - used in case you're running it from somewhere else. - - ~hugo~ Compile the website into static files located in the - =public= directory. - - ~rsync -va --progress --rsync-path="/usr/bin/openrsync" public/ - blog@lambda.cx:/var/www/htdocs/lambda.cx/blog~ This one is bigger - so I'll break it down. - - =rsync= A command that synchronizes files between two directories - - =-v= Be verbose, this is optional but I like it - - =-a= Stands for "archive": copy recursively, keep - permissions, etc. See the =rsync= man page if you're curious. - - =--progress= Show progress, also optional - - ~--rsync-path="/usr/bin/openrsync"~ This line is very important - for OpenBSD servers. OpenBSD has its own =rsync= implementation - called =openrsync=. Without this argument, =rsync= will connect - to the server, see that the =rsync= command doesn't exist, and - fail. - - =public/= Specify which folder we want to sync. The trailing - =/= is important. Without it =rsync= will copy the folder - instead of the folder's contents, which is what we want. - - =blog@lambda.cx:/var/www/htdocs/lambda.cx/blog= Login to user - =blog= on server =lambda.cx=, syncing files with the - =/var/www/htdocs/lambda.cx/blog= directory. - - The reason to use =rsync= here instead of something like =scp= is - that =rsync= won't upload files it doesn't need to, so if 3/4 of - the files didn't change when you updated the blog, it won't waste - time re-uploading them. diff --git a/content/posts/letsencrypt-on-openbsd/index.org b/content/posts/letsencrypt-on-openbsd/index.org deleted file mode 100644 index 556404b..0000000 --- a/content/posts/letsencrypt-on-openbsd/index.org +++ /dev/null @@ -1,122 +0,0 @@ -#+TITLE: Let's Encrypt on OpenBSD -#+DATE: 2020-06-16T22:56:27-04:00 -#+DRAFT: false -#+DESCRIPTION: Setting up acme-client on OpenBSD -#+TAGS[]: openbsd httpd letsencrypt acme-client -#+KEYWORDS[]: openbsd httpd letsencrypt acme-client -#+SLUG: -#+SUMMARY: - -#+ATTR_HTML: :alt Let's Encrypt OpenBSD -#+ATTR_HTML: :title Let's Encrypt OpenBSD -[[file:openbsd%20letsencrypt.png]] - -So I have an OpenBSD server serving a static website using -=httpd=. I've been thinking for a while I should add an SSL -certificate, but never got around to it because it was just a small -hobby website and it didn't require any real attention. - -Today while watching one of the OpenBSD tutorials at BSDCan, I thought -it was finally time. Since configuring everything else in OpenBSD is -so easy, this must be easy too, right? - -These were the only changes I had to make to my =httpd.conf= to get -=acme-client= to work. This is described in the =acme-client= man -page. -#+BEGIN_SRC diff ---- httpd.conf -+++ httpd.conf.new -@@ -1,4 +1,19 @@ - server "lambda.cx" { - listen on * port 80 - root "/htdocs/lambda.cx" -+ location "/.well-known/acme-challenge/*" { -+ root "/acme" -+ request strip 2 -+ } - } -#+END_SRC - -After that, I reloaded =httpd= with ~rcctl reload httpd~ - -I then copied the example config from =/etc/examples/acme-client.conf= -to =/etc/acme-client=. This is what the modifications to the example I -made look like. - -#+BEGIN_SRC diff ---- acme-client.conf -+++ acme-client.conf.new -@@ -1,19 +1,19 @@ - # - # $OpenBSD: acme-client.conf,v 1.2 2019/06/07 08:08:30 florian Exp $ - # - authority letsencrypt { - api url "https://acme-v02.api.letsencrypt.org/directory" - account key "/etc/acme/letsencrypt-privkey.pem" - } - - authority letsencrypt-staging { - api url "https://acme-staging-v02.api.letsencrypt.org/directory" - account key "/etc/acme/letsencrypt-staging-privkey.pem" - } - --domain example.com { -- alternative names { secure.example.com } -- domain key "/etc/ssl/private/example.com.key" -- domain full chain certificate "/etc/ssl/example.com.fullchain.pem" -+domain lambda.cx { -+ # alternative names { www.lambda.cx } -+ domain key "/etc/ssl/private/lambda.cx.key" -+ domain full chain certificate "/etc/ssl/lambda.cx.fullchain.pem" - sign with letsencrypt - } -#+END_SRC - -It's a pretty small change. I have the alternative name line commented -out because I only have =lambda.cx= pointing at my server and not -=www.lambda.cx=. Although if I did I would un-comment it. I could also -add sub-domains like =sub.lambda.cx= in that area separated by a -space. - -After that I just had to run ~acme-client -v lambda.cx~ (-v for -verbosity) and it generated the certificates. - -Then I added a =crontab= entry (using =crontab -e=) to run once a day -at a random time and reload =httpd=. - -#+BEGIN_SRC -~ ~ * * * acme-client lambda.cx && rcctl reload httpd -#+END_SRC - -Finally to use the new certificates I added the following lines to my -=httpd.conf=. - -#+BEGIN_SRC diff ---- httpd.conf -+++ httpd.conf.new -@@ -1,8 +1,21 @@ - server "lambda.cx" { - listen on * port 80 - root "/htdocs/lambda.cx" - location "/.well-known/acme-challenge/*" { - root "/acme" - request strip 2 - } - } -+ -+server "lambda.cx" { -+ listen on * tls port 443 -+ tls { -+ certificate "/etc/ssl/lambda.cx.fullchain.pem" -+ key "/etc/ssl/private/lambda.cx.key" -+ } -+ root "/htdocs/lambda.cx" -+ location "/.well-known/acme-challenge/*" { -+ root "/acme" -+ request strip 2 -+ } -+} -#+END_SRC - -I reloaded httpd with ~rcctl reload httpd~ and that was it, working -certificate! diff --git a/content/posts/letsencrypt-on-openbsd/openbsd letsencrypt.png b/content/posts/letsencrypt-on-openbsd/openbsd letsencrypt.png deleted file mode 100644 index f805be0..0000000 Binary files a/content/posts/letsencrypt-on-openbsd/openbsd letsencrypt.png and /dev/null differ diff --git a/content/posts/openvpn-issues-openbsd-6.7/index.org b/content/posts/openvpn-issues-openbsd-6.7/index.org deleted file mode 100644 index a9c34bd..0000000 --- a/content/posts/openvpn-issues-openbsd-6.7/index.org +++ /dev/null @@ -1,72 +0,0 @@ -#+TITLE: Issues with OpenVPN on OpenBSD 6.7 -#+DATE: 2020-06-14T14:08:06-04:00 -#+DRAFT: false -#+DESCRIPTION: -#+TAGS[]: openvpn openbsd libressl -#+KEYWORDS[]: -#+SLUG: -#+SUMMARY: - -#+ATTR_HTML: :alt No connection to ProtonVPN from OpenBSD -#+ATTR_HTML: :title No connection to ProtonVPN from OpenBSD -[[file:openbsd%20protonvpn%20no%20connection.png]] - -I have an OpenBSD VPN gateway I use to send all traffic it receives -over a VPN connection, and I noticed that no traffic was going through. - -I'd been using ProtonVPN as my provider for months prior to this -without any issues, so it was very confusing when it stopped working. - -No matter which VPN profile I used from ProtonVPN, it always gets -stuck after the step =TLS: Initial packet from -[AF_INET]XXX.XXX.XXX.XXX:YY=. Regardless of whether I tried the -individual server profiles, country profiles, free, or plus profiles. - -I tried starting openvpn with maximum verbosity. Everything launched -exactly as it should, until it gets to the TLS handshake, where it -failed to get a response. - -#+BEGIN_SRC -Sun Jun 14 15:37:22 2020 us=577519 UDP link local: (not bound) -Sun Jun 14 15:37:22 2020 us=577532 UDP link remote: [AF_INET]XXX.XXX.XXX.XXX:YYYY -Sun Jun 14 15:37:22 2020 us=577650 UDP WRITE [86] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 pid=[ #1 ] [ ] pid=0 DATA len=0 -Sun Jun 14 15:37:22 2020 us=739355 UDP READ [98] from [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_HARD_RESET_SERVER_V2 kid=0 pid=[ #1 ] [ 0 ] pid=0 DATA len=0 -Sun Jun 14 15:37:22 2020 us=739517 TLS: Initial packet from [AF_INET]XXX.XXX.XXX.XXX:YYYY, sid=19fe5aac 2d00f4aa -Sun Jun 14 15:37:22 2020 us=739658 UDP WRITE [94] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_ACK_V1 kid=0 pid=[ #2 ] [ 0 ] -Sun Jun 14 15:37:22 2020 us=739798 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this -Sun Jun 14 15:37:22 2020 us=739900 UDP WRITE [331] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_V1 kid=0 pid=[ #3 ] [ ] pid=1 DATA len=245 -Sun Jun 14 15:37:24 2020 us=832019 UDP WRITE [331] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_V1 kid=0 pid=[ #4 ] [ ] pid=1 DATA len=245 -Sun Jun 14 15:37:29 2020 us=32189 UDP WRITE [331] to [AF_INET]XXX.XXX.XXX.XXX:YYYY: P_CONTROL_V1 kid=0 pid=[ #5 ] [ ] pid=1 DATA len=245 -#+END_SRC - -It just kept repeating the write until it timed out after 60 -seconds. It was like this for every country, on every port. Even using -the TCP profiles, but instead there the connection would get reset -almost immediately instead of timing out. - -I tried several free VPNs I found online just to compare, and all of -them worked without issue. This problem has only happened for me with -ProtonVPN servers. - -I tried connecting using both my desktop machine, and an Ubuntu VM, -both of which were able to connect without issue. The problem wasn't -with the account itself. - -I tried using another OpenBSD VM on my network, and the result was the -same as the VPN gateway, a timeout. I even spun up a fresh OpenBSD VM -in Vultr to see if the issue persisted on a new install in a different -network. The issue was still there. - -I was sure to check that the system clocks were correct on each -machine, and also tried commenting out all lines in the VPN profile -that weren't strictly required to make the connection, like mtu and -compression settings. - -As a final attempt, I tried installing OpenVPN with =mbedtls=. For all -my previous experiments, I had been using the default openvpn package, -which uses OpenBSD's LibreSSL. That time it worked perfectly. - -It occurred to me that this had been the first time I'd checked up on -the VPN gateway since updating to OpenBSD 6.7. I guess something about -a recent LibreSSL update has broken a feature OpenVPN relies on in -certain situations. diff --git a/content/posts/openvpn-issues-openbsd-6.7/openbsd protonvpn no connection.png b/content/posts/openvpn-issues-openbsd-6.7/openbsd protonvpn no connection.png deleted file mode 100644 index f4fbe9b..0000000 Binary files a/content/posts/openvpn-issues-openbsd-6.7/openbsd protonvpn no connection.png and /dev/null differ diff --git a/content/posts/pcengines-comparison/index.org b/content/posts/pcengines-comparison/index.org deleted file mode 100644 index a80f773..0000000 --- a/content/posts/pcengines-comparison/index.org +++ /dev/null @@ -1,96 +0,0 @@ -#+TITLE: PC Engines APU Comparison -#+DATE: 2020-06-17T00:50:06-04:00 -#+DRAFT: false -#+DESCRIPTION: Comparison between PC Engines APU machines -#+TAGS[]: hardware pcengines -#+KEYWORDS[]: hardware pcengines -#+SLUG: -#+SUMMARY: - -#+ATTR_HTML: :alt PC Engines Comparison -#+ATTR_HTML: :title PC Engines Comparison -[[file:pc%20engines%20vs.png]] - -I've been looking at the [[https://www.pcengines.ch/apu2.htm][PC Engines APU]] line for a while. They're a -line of medium size single board PCs with a DB9 serial connector and -no VGA port. They also have gigabit Ethernet. Because of this they're -often used as firewall machines. - -I want to get one and use it as either the home router, or an -experimental server to mess around with. Quite a few OpenBSD folks use -them and recommend them as OpenBSD router and server hardware. They -aren't too expensive, have decent specs and a small physical -footprint. - -One thing that's always confused me was the naming scheme, which is a -little confusing at first. Initially there was the APU, then the APU2, -as described on their site, which makes sense. Then the APU model -numbers get a little confusing. They come in several variants, I'll -list them here for context. - -#+CAPTION: Taken from the PC Engines website -#+BEGIN_SRC - apu2d0 (2 GB DRAM, 2 i211AT NICs) - apu2e2 (2 GB DRAM, 3 i211AT NICs) - apu2e4 (4 GB DRAM, 3 i210AT NICs) - apu3c2 (2 GB DRAM, 3 i211AT NICs, optimized for 3G/LTE modems) - apu3c4 (4 GB DRAM, 3 i211AT NICs, optimized for 3G/LTE modems) - apu4d2 (2 GB DRAM, 4 i211AT NICs) - apu4d4 (4 GB DRAM, 4 i211AT NICs) - #+END_SRC - -What do the letters between the numbers mean? What is the significance -of the numbers in the first place? - -Let's take the =apu3c2= as an example. The =3= here means it's version -3 of the APU board. The APU 1 is no longer sold, so it's left out of -the list on the PC Engines website, along with most of their -re-sellers. The letter =c= is the revision of that board. It's pretty -much irrelevant for comparison. You'll want to get the most recent -one, as listed on the PC Engines website. The revisions are mostly -minor tweaks to the board. The final number, =2=, is the number of gigabytes -of RAM (in most cases). - -The APU2E2 boards have a single mSATA slot, and a regular SATA -connector, two mPCIe slots, and a SIM tray. It comes with 2 external -USB 3.0 type A ports, 2 internal USB 2.0 ports (header only), an SD -card slot, and a GPIO header. It also has 3 Intel i211AT gigabit -NICs. Using the SIM tray will remove the ability to use one of the two -mPCIe slots. - -So what are the major differences between version numbers? They all -use the same CPU, the quad core AMD Embedded G series GX-412TC running -at 1 GHz. Most other components on the boards are the same as -well. I'll give a description of the differences relative to the APU2 -board. - -The APU2D0 is the same as the regular APU2, but without a GPIO -headers, and it has only 2 NICs. - -The APU3 has an extra SIM tray, and the first slot, which is mSATA -only on the APU2, can be used as either an mSATA or USB 3G/LTE Modem -slot. The second slot can only be used for a modem. It also lets you -do GPIO tray swapping and failover, so you can use only a single modem -for 2 SIMs. - -The APU4D2 has 4 NICs instead of 3, and the second slot is modem -only. It has 2 GB of RAM. - -The APU4D4 is the same as the APU4D2, except it has dual SIM trays. It -has 4 GB of RAM. - -I found a lot of this information wasn't really obvious to upon first -reading the PC Engines website, as there's no direct comparison -between the board, and the site relies on you flipping between pages -to find the details. The re-seller website I looked at also didn't -elaborate on the differences. - -| Board | RAM | Slot 1 | Slot 2 | Slot 3 | Ethernet Controllers | SIM Slots | USB 2.0 | Headers | -|--------+------+-------------+-------------+--------+----------------------+-----------+---------+---------| -| [[https://www.pcengines.ch/apu2d0.htm][apu2d0]] | 2 GB | mSATA | mPCIe/Modem | mPCIe | 2 i211AT | 1 | 2 | No | -| [[https://www.pcengines.ch/apu2e2.htm][apu2e2]] | 2 GB | mSATA | mPCIe/Modem | mPCIe | 3 i211AT | 1 | 2 | Yes | -| [[https://www.pcengines.ch/apu2e4.htm][apu2e4]] | 4 GB | mSATA | mPCIe/Modem | mPCIe | 3 i210AT | 1 | 2 | Yes | -| [[https://www.pcengines.ch/apu3c2.htm][apu3c2]] | 2 GB | mSATA/Modem | Modem | mPCIe | 3 i211AT | 2 | 4 | Yes | -| [[https://www.pcengines.ch/apu3c4.htm][apu3c4]] | 4 GB | mSATA/Modem | Modem | mPCIe | 3 i211AT | 2 | 4 | Yes | -| [[https://www.pcengines.ch/apu4d2.htm][apu4d2]] | 2 GB | mSATA | Modem | mPCIe | 4 i211AT | 1 | 2 | Yes | -| [[https://www.pcengines.ch/apu4d4.htm][apu4d4]] | 4 GB | mSATA/Modem | Modem | mPCIe | 4 i211AT | 2 | 2 | Yes | diff --git a/content/posts/pcengines-comparison/pc engines vs.png b/content/posts/pcengines-comparison/pc engines vs.png deleted file mode 100644 index 88c715a..0000000 Binary files a/content/posts/pcengines-comparison/pc engines vs.png and /dev/null differ -- cgit v1.2.3