unadulterated faff

Compiling st for Ubuntu 18.04

Recently I wanted to compile Luke Smiths’ fork of st, the Suckless simple terminal. The compilation setup is tailored to Arch out-of-the-box, so I made some changes to get everything to work on Ubuntu 18.04.

Notably, I needed to:

1. Install a number of dependencies using apt-get.
2. Compile and install HarfBuzz, the text shaping engine used by st.

The main roadblock was that the compilation instructions from the HarfBuzz repository use meson and running meson build && meson test -Cbuild resulted in some meson lexer error that wasn’t listed in the GH issues. Given how little I know about meson, I wasn’t ecstatic about figuring out the issue. Thankfully, HarfBuzz supports the autoconf / automake toolchain as well (though it is being deprecated in favor of meson), so I used that instead.

For more info on building HarfBuzz, see here.

# Clone the required repos:
cd Workspace
git clone https://github.com/LukeSmithxyz/st.git
git clone https://github.com/harfbuzz/harfbuzz.git

# Now use apt-get to install the packages you'll need.
# Only one of the two commands below is required:
# If you use autoconf & automake like I did, the meson and ragel packages are unnecessary.
# sudo apt-get install -y meson ragel
# If instead you go with the meson build, autoconf and automake are unnecessary.
sudo apt-get install -y autoconf automake
# These packages are what you will need in either case:
sudo apt-get install -y libtool pkg-config gtk-doc-tools gcc g++ libfreetype6-dev libglib2.0-dev libcairo2-dev libx11-dev libxft-dev libxext-dev

# Compile HarfBuzz:
cd harfbuzz
# To use the meson toolchain run:
# meson build && meson test -Cbuild
# To use the autoconf // automake toolchain instead run:
./autogen.sh
make && sudo make install

# Here, double check that the required .so files for HarfBuzz were installed into /usr/local/include/harfbuzz
ll /usr/local/include/harfbuzz
# This should produce a list of several files with names like hb-*.h
# If it doesn't, check to see if they were installed in /usr/include/harfbuzz
# Failing that, you will need to examine the log files in the harfbuzz directory to determine where things
# went awry.

# Now we can compile `st`:
cd ../st
# Apply the edits mentioned below:
vim config.mk
make
# Run the resulting binary:
./st

The edits you need to make to config.mk are as follows:

X11INC = /usr/include/X11R6
X11LIB = /usr/lib/X11R6

Changing where screenshots are saved on macOS

By default, macOS saves screenshots to the Desktop; if you are like me and like to keep your desktop empty, this can be pretty irritating.

The irritation finally hit a critical mass today and I decided to figure out how to solve the problem. I bumped into this post on the Apple forums, which has some very clear instructions, but I will reproduce the critical bit here.

Use the defaults tool to set the screenshot location to an arbitrary path. defaults has a few possible actions, of which a few are relevant to us:

read -- Show all of the current defaults values for all applications. If you specify an application using its fully-qualified domain name (eg `com.foo.bar` or `org.baz.quux`), only that application's defaults will be printed out.

delete <domain> -- Nukes all defaults for an application.
delete <domain> <key> -- Delete a specific key-value pair for an application.

write <domain> <key> <value> -- Sets a key-value pair for a domain.

This latter option is what we need: for the domain used by the screenshot tool (com.apple.screencapture), set the location key to the desired path.

defaults write com.apple.screencapture location /Users/eindiran/Pictures/Screenshots

If you haven’t created the folder you plan to use, you can create it in Finder or run mkdir -p /my/new/path

Setting up a browser to improve privacy

The modern internet makes avoiding tracking and preserving any of your privacy into a Sisyphean task; many people see this and just give up any hope of having privacy at all. Something that feeds into this is that many privacy advocates encourage taking an all-or-nothing approach: delete all of your accounts with Google, Facebook, etc. and only connect to the internet via Tails and Tor on public wifi.

While these are measures that will protect your privacy, and it may be worthwhile learning how to use these tools, you can significantly reduce how much you are tracked by setting up your browser differently. The goal of this post is to show how you can quickly setup a browser that limits how much you are tracked.

Setting up your browser

In this section, I will be describing how to do everything on a Debian-based Linux distribution, but on Windows, macOS, and *nix systems, this process should be generally the same: in big strokes, you are cleaning out all of your existing data except the stuff you explicitly save somewhere, and then installing 2 fresh browsers - Firefox (what you’ll use normally) and Chrome (which you will use as your backup for when Firefox is having trouble).

To start with, make sure to save any data you want to keep. For example, get a copy of your bookmarks and history. Then place whatever you kept somewhere for safekeeping.

Next, uninstall your browsers and delete the directories where they keep their info. On Linux, check ~/.config/<browser name>:

sudo apt-get purge firefox google-chrome-stable chromium-browser
rm -rf ~/.config/google-chrome
rm -rf ~/.config/chromium
rm -rf ~/.mozilla/firefox/

Then install a fresh copy of Firefox:

sudo apt-get install firefox

Set update-alternatives for your browser to make Firefox your default browser:

update-alternatives --set gnome-www-browser "$(which firefox)"

Open up Firefox, then head over to the preferences. Go through the settings and set things up to your taste. There are a few you should definitely turn on though:

• Set Enable Container Tabs to be checked.
• Turn off Check your spelling as you type.
• Set your default search engine to be DuckDuckGo and delete all of the other search engine options.
• Change your Search Suggestion preferences to be as restrictive as you are willing to tolerate.
• Turn Enhanced Tracking Protection to Custom with cookies checked and set to All third-party cookies, and all other boxes checked.
• Always send a “Do Not Track” signal, even though nobody actually respects it.
• Set Firefox to delete cookies and site data when closed: then add all the sites to the whitelist that you want to stay logged in for.
• Turn off saved logings/passwords, autofill for forms/addresses, etc.
• For the permissions panel, set everything to be as restrictive as you are willing to tolerate.
• Turn on Deceptive Content and Dangerous Software Protection.
• For the question When a server requests your personal certificate select Ask you every time.

Next, we will install all of the addons we want to help us out. Open the Firefox addons page and install the following addons:

1. uBlock Origin - This is our main ad blocker and most important addon. It blocks a ton of tracking and ads; even by itself it is a HUGE quality of life improvement over vanilla.
2. Decentraleyes - This will block CDNs like Google Hosted Libraries, while hosting the data locally to prevent sites from breaking.
3. NoScript - This allows us to block Javascript, which is going to be one of our biggest sources of nastiness. We can use this to allow some sites to use Javascript, which is a big improvement over blocking all Javascript.
4. Firefox Multi-Account Containers - Very useful tool for sandboxing a site. Then the site can’t peep on your other cookies etc, that aren’t included in the container.
5. Privacy Badger - Optional, if you are using everything else above. Blocks trackers, mopping up some of those missed by our other tools above.
6. DuckDuckGo Privacy Essentials - Optional. This has the excellent feature of showing a privacy score for sites, which gives you a decent idea of how much the site you’re using respects your privacy.
7. HTTPS Everywhere - Optional. Makes sure that you always use the HTTPS version of a site when available.
8. SmartReferer - Optional. Prevents shenanigans regarding referers.

Once you’ve installed each of the addons, visit some sites and see what breaks. You will need to start whitelisting things, which is very tedious at first but after a few days becomes very managable from that point forward. Stick with it and you’ll really benefit in the long term. But, you will sometimes still hit errors, which brings me to my next topic.

Troubleshooting errors

Because lots of sites rely on Javascript, bad referer handling practices, trackers, ads, etc, using this setup will break some sites. Sometimes these will be sites you need to use, so you will need to be able to troubleshoot these problems and having a good mental model of what the errors are helps immensely.

These are the steps I generally take, in order:

1. Enable Javascript by clicking on the NoScript icon, then selecting the Disable restrictions for this tab button, then refresh the page. If you use this page regularly, it may be worth figuring out which domains you should add to your NoScript whitelist to allow it to work.
2. Check the console. Right click Inspect Element and open the second tab, titled Console. There is often a description of what errors are happening, which can inform your next steps.
3. Try turning off SmartReferer and refreshing the page. Some sites use subdomains that SmartReferer doesn’t like (eg the AWS Console), so it can lead to a dead white page. Note that this problem often leaves telltale signs in the console, so once you’ve found it, it may be worth checking what is shown there.
4. If none of the above steps worked or gave you actionable information, temporarily disable each addon one-by-one to determine which one is responsible for the breakage. You can then whitelist the site in that addon.
5. If the site still doesn’t work with Firefox when all addons are turned off, it may just be that only Chrome/Safari/Edge is supported. Consider whether you actually want to use/support this site, given that it is misbehaved and probably doesn’t respect your privacy at all: if you do still want to use it, fire up Chrome.

Ultimately, you will need to find what works for you, and decide if you want to use each of the addons.

Using a secondary browser

I highly recommend that once you have a setup you like, install a secondary browser (probably Chrome) which you keep totally vanilla. Only open this browser up as a last resort, when you can’t use Firefox with all of the addons turned off.

Downsides to this approach

One big downside is that the first week will feel pretty bad: lots of sites you use will barely work or not work at all, and it will seem like you need to constantly be tweaking your setup to get anything to work. But this feeling will pass once you’ve gotten used to your setup and your whitelists are well populated with the main sites you use.

Another potential downside: because this is a non-standard setup, this configuration could be used to fingerprint your browser. To check out how this works, visit the EFF’s Panopticlick site. But the vast majority of people tracking you will be doing it in a way that is easily foiled by this setup, so unless your threat model warrants something stronger, the upsides significantly outweigh the downsides.

Installing Python3.8 on Ubuntu 18.04

By default Ubuntu 18.04 includes Python 3.6. If you want a later version of Python (namely 3.7, 3.8 or 3.9), it is possible to install them using an apt PPA. PPAs, or ‘Personal Package Archives’, are additional repositiories that apt can use to search for packages, but they can contain packages that haven’t undergone enough testing and validation to end up in one of the main repos: this means that they may not receive timely updates, they might have security problems, they might be seriously broken, etc. Consider this a caveat emptor.

The PPA you’ll need to install a later version of Python 3 is the deadsnakes PPA; deadsnakes has a copy of each version of Python from 2.3 through 3.9, so you can grab earlier versions as well.

To begin, add the PPA and run update to grab the package list.

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update

Then just install the version of Python you need:

sudo apt install python3.7
sudo apt install python3.8
sudo apt install python3.9

Note that it won’t be installed over your system copy of Python 3, so you’ll need to invoke it with python3.X rather than python or python3.

2 + 2 = 5: Adventures with the CPython integer cache

An interesting quirk of CPython (one which has caused debugging woes for many a Python-newcomer) is that the CPython interpreter preallocates the integers -5 through 256 inclusive in a special area of memory, often called the “integer cache” or the “small integer cache”. What this means is that every time you assign a variable to an integer in that range, the variable is a reference to that integer’s location in the cache. We can verify this by using getrefcount from the sys package:

>>> import sys
>>> sys.getrefcount(256) # Get the baseline count for 256
23
>>> x = 256              # Increment the count by 1
>>> sys.getrefcount(256)
24
>>> y = 256              # Increment it by 2
>>> z = 256
>>> sys.getrefcount(256)
26
>>> sys.getrefcount(257) # Now get the baseline for an integer not in the cache
3
>>> a = 257
>>> sys.getrefcount(257) # The count didn't go up!
3
>>> b = 257
>>> sys.getrefcount(257)
3

Each time we assign a variable to an integer in range, the reference count for the integer goes up. But when we’re assigning a variable to an integer outside the range, it has no effect on the reference count of the integer since each variable points to a distinct location in memory.

Stated in the simplest terms, when you set x = 1 and y = 1, both x and y point to the same place in memory. We can see that that is true, using id and is, since id shows memory address and is shows object equality:

x = 256
y = 256
x == y          # True
x is y          # True
id(x) == id(y)  # True


a = 257
b = 257
a == b          # True
a is b          # False!
id(a) == id(b)  # False!

The integer cache is used by the CPython interpreters for both Python 2 and 3 (though I suppose we are now not supposed to acknowledge that 2 even exists), but today we’ll just be using the CPython 3.6 interpreter. Take a look at the official docs for the Python C API: it points out that the integer cache is just implemented as an array of integer objects, so in principle it is possible to change the value of each integer in the array. From the docs:

The current implementation keeps an array of integer objects for all integers between -5 and 256, when you create an int in that range you actually just get back a reference to the existing object. So it should be possible to change the value of 1. I suspect the behaviour of Python in this case is undefined. :-)

This made me curious: what does happen when you change the value of the integers stored in the integer cache? What kinds of aberrant and unexpected behavior will we see? What if we changed all of them?

Part One: Figuring out the plan of attack

If we want to figure out the answers to the above questions, we should start by running some experiments. I generally like just throwing things at the wall and seeing what sticks, so let’s just blindly dive in.

Experiment 1 – The size of values in the integer cache

My first experiment was to use id to examine the address of each int in the integer cache:

>>> id(1)
10914496
>>> id(2)
10914528
>>> id(2) - id(1)
32
>>> id(3) - id(2)
32
>>> id(4) - id(3)
32

You’ll notice that they are each offset by 32 bits, which we should expect since they are each a long.

Experiment 2 – Abusing ctypes

CPython is implemented in C under-the-hood, and provides the package ctypes to interact with a lot of interpreter internals from within the interpreter. Let’s abuse some ctypes tools to see what we can do. The bits from ctypes that we need:

• POINTER - a factory function that takes a type and creates a pointer of that type. For example, POINTER(c_ulong) creates a ctypes unsigned long pointer.
• addressof - a function that returns the address of a ctypes object. It behaves in the same way that id does, but only works for ctypes objects.
• cast - a function that takes a memory address and a ctypes pointer type and returns a pointer of the specified type to the address. Abusing this function is one of the easiest ways to get CPython to segfault.

Our next step will be to use cast to print the value of a memory location as the ctypes c_long type, which is a 32-bit integer.

>>> from ctypes import *
>>> print(cast(id(1), POINTER(c_long))[0])
1621
>>> print(cast(id(2), POINTER(c_long))[0])
525

Okay. That is definitely not what we expected to see (well, at least it wasn’t what I expected to see - maybe your mental model of the CPython interpreter is much better than mine). Let’s back up for a second, and take a look at what’s going on here with the cast call here. I constructed this example, which demonstrates that in principle using cast like we did above should work:

>>> from ctypes import *
>>> x = (c_ulong * 5)() # Create an array of unsigned longs
>>> x[0] = 18
>>> x[1] = 19
>>> x[2] = 33
>>> x
<__main__.c_ulong_Array_5 object at 0x7fd97b051c80>
>>> cast(x, POINTER(c_ulong))[0]
18
>>> cast(x, POINTER(c_ulong))[1]
19
>>> cast(x, POINTER(c_ulong))[2]
33

Experiment 3 – Looking for an offset

So in principle we are doing the right thing with cast; perhaps the issue is that we have the wrong offset. To figure out what the right offset might be, I took a look at each value between 0 and 32, and tried to use that value as the offset:

>>> start = 0
>>> end = 32
>>> for h in range(start, end + 1):
...     print("{}: {}".format(h, cast(id(1) + h, POINTER(c_long))[0]))
...
0: 850
1: -9223372036854775805
2: -3350678122763649024
3: -7146790396171911168
4: 44140444052881408
5: 172423609581568
6: 673529724928
7: 2630975488
8: 10277248
9: 72057594037968081
10: 281474976710812
11: 1099511627776
12: 4294967296
13: 16777216
14: 65536
15: 256
16: 1
17: 72057594037927936
18: 281474976710656
19: 1099511627776
20: 4294967296
21: 16777216
22: 65536
23: 256
24: 1
25: 8142508126285856768
26: 31806672368304128
27: 124244813938688
28: 485331304448
29: 1895825408
30: 7405568
31: 28928
32: 113

So there are two 1 values in there, at offset 16 and offset 24. Let’s take a look at 2 and 3 and hopefully a pattern will emerge:

>>> for h in range(start, end + 1):
...     print("{}: {}".format(h, cast(id(2) + h, POINTER(c_long))[0]))
...
0: 115
1: -9223372036854775808
2: -3350678122763649024
3: -7146790396171911168
4: 44140444052881408
5: 172423609581568
6: 673529724928
7: 2630975488
8: 10277248
9: 72057594037968081
10: 281474976710812
11: 1099511627776
12: 4294967296
13: 16777216
14: 65536
15: 256
16: 1
17: 144115188075855872
18: 562949953421312
19: 2199023255552
20: 8589934592
21: 33554432
22: 131072
23: 512
24: 2
25: 3746994889972252672
26: 14636698788954112
27: 57174604644352
28: 223338299392
29: 872415232
30: 3407872
31: 13312
32: 52
>>> for h in range(start, end + 1):
...     print("{}: {}".format(h, cast(id(3) + h, POINTER(c_long))[0]))
...
0: 54
1: -9223372036854775808
2: -3350678122763649024
3: -7146790396171911168
4: 44140444052881408
5: 172423609581568
6: 673529724928
7: 2630975488
8: 10277248
9: 72057594037968081
10: 281474976710812
11: 1099511627776
12: 4294967296
13: 16777216
14: 65536
15: 256
16: 1
17: 216172782113783808
18: 844424930131968
19: 3298534883328
20: 12884901888
21: 50331648
22: 196608
23: 768
24: 3
25: 4611686018427387904
26: 18014398509481984
27: 70368744177664
28: 274877906944
29: 1073741824
30: 4194304
31: 16384
32: 64

So it looks like our offset might be 24/0x18. Let’s test that for each value:

>>> for x in range(-5, 256 + 1):
...     y = cast(id(x) + 0x18, POINTER(c_long))[0]
...     print("x: {0} --> {1}".format(x, y))
...     assert abs(x) == y
...
x: -5 --> 5
x: -4 --> 4
x: -3 --> 3
x: -2 --> 2
x: -1 --> 1
x: 0 --> 0
x: 1 --> 1
x: 2 --> 2
x: 3 --> 3
x: 4 --> 4
x: 5 --> 5
...
x: 254 --> 254
x: 255 --> 255
x: 256 --> 256

Experiment 4 – Changing a value in the integer cache

Aha! It looks like we’ve found the correct offset. Now let’s just try dumbly assigning a value to a random integer in that range:

>>> cast(id(13) + 0x18, POINTER(c_long))[0] = 1
>>> 13 + 1
2

Awesome, now we’re getting somewhere. Let’s see if we can finally make Big Brother’s math correct:

>>> cast(id(5) + 0x18, POINTER(c_long))[0] = 4
>>> 2 + 2 == 5
True

There are still a lot of our original questions left unanswered which we’ll try to tackle in an upcoming post. More on the Python integer cache to come.