talkingquickly

This is a page I update periodically with key takeaways from non-fiction books I've read. My current top 25 are:

1. Four thousand weeks
2. Fooled by randomness
3. Atomic Habits
4. Make Time
5. Thanks for the feedback
6. Radical Candor
7. High output management
8. The 4 hour work week
9. The 4 hour body
10. The hard thing about hard things
11. The lean startup
12. Deep Work
13. Digital Minimalism
14. The power of now
15. Why we sleep
16. In defence of food
17. The tipping point
18. Breath
19. Outlive
20. Getting things done
21. Algorithms to live by
22. Thinking in systems
23. Zen and the art of motorcycle maintenance
24. The 48 laws of power
25. Inspired
26. The dip

It's mainly a tool for me to skim through and remind myself of what I've read as a way to jog my thinking. The summaries are a mixture of notes I wrote when reading them, reminders from Shortform summaries and what I remember. So they've been heavily filtered by my interpretation and what I was thinking about at the time I read them. They are almost certainly not accurate summaries of the books themselves!

more

Four thousand weeks

This is the book that gave me a framework for carving out dedicated, non negotiable time usually early in the morning for an ongoing passion project. For a long time I used the “make time” app for this.

Four thousand weeks focuses on the shortness of life and how we spend it. It drives home that the deciding factor in how we spend our life is where and how we direct our attention.

One of the quotes that has never left me is about how apt the idea of "spending" your time is, because once each moment is spent, you can never have it back:

Pay attention, because you are paying with your life

It emphasises how time will pass no matter what, and so it is our decisions about where to direct our attention that govern the quality of our life.

The core thread of the book is around the extent to which you really control your time. In essence that we never have full control over our time, or at least we will never be able to do all of the things we want. There are probably a variety of reasons for this, not least that as we get more efficient, we come up with more things we want to do. So we will never get to the end of the list.

This resonates with one of the most important lessons I learned from my father. That many people imagine if they could just have a day to get to the end of their todo list, they'd be caught up and "on top of things". But in reality this would be true for mere hours until the todo list started to grow again. So the real skill in life is doing the right things from the list and learning to live peacfully with the fact it will never end.

The book turns to some practical considerations for making the most of the little time we have:

1. Force time for the things that matter, never wait for time to "open up". Because it never will.
2. Limit your work in progress projects. Probably to 3-4.
3. Become comfortable with and embrace discomfort. Especially the type of discomfort that might cause you to deviate from your project. The main approach suggested is noticing the distraction and discomfort and directing constantly increasing attention to it rather than shying away from it.
4. Stop expecting the future to unfold in a particular away. In particular reflect on how little of your life to date your eally controlled and stop expecting a far greater degree of control over the future.
5. Develop patience for how long things really take. An interesting tactic here is time boxing heavily and refusing to allow yourself more than that amount of time to work on something. Whenever you run out of time, your forced to become a little more comfortable with the feeling of impatience.
6. Align your free time with your friend. As someone who obsessively tries to control my schedule, this one came from left field. The goal being to align your schedule to maximise the chances you can spend time with those you love.

The second part of the book delves more deeply into the idea of life being finite. This comes with the - to me at least - calming sentiment that you can never have all of the experiences you want, because every choice you make implicitly takes the space of something else. Because it is fundamentally finite, the thing that matters is choosing things that matter, not wasting time on todo lists.

The book suggests four tactics in particular:

1. Make and strongly commit to life choices. The essence of this is that since you'll never be able to do everything, you'll be happier if you commit to something and do it well. The underlying principle is that more happiness is created by committing and doing well than by keeping options open.
2. Focus on the present not the future. This hit quite hard, that it's tempting to spend lots of time on actions with future payoffs, to control the future. But generally these payoffs are highly uncertain, but the payoffs now are much clearer. An example would be working on a marketing campaign for the future vs going outside to enjoy the good weather. The happiness from the good weather is certain, the payoff from the marketing campaign is very much not. I didn't interpret this as not investing in the future, just that many people - me included - have a tendancy to over-invest in the future, at the expense of realising a return on the present.
3. Incorporate purposeless time. I loved this one. In a world where it's fashionable to try and turn every hobby into a profit making "side hustle", the emphasis here is on the importance of doing things purely for the sake of doing them, with no expectation of a return.
4. Don't live only for changing the world, because you probably won't. Nobody has that much impact on the world in the grand scheme of things. Humanities impact will be insignificant on many timescales. Once you let go of a need to make a grand change to the world, you open yourself up to being able to make the little changes which are possible and actually matter.

On a tangent, this is the book that led me to have a poster on my wall where I mark off the weeks, counting down to 4000. Tosome people this seems morbid. For me it aligns nicely with the parts of Stoicism I relate to, acting as a reminder to enjoy each moment.

Reading list

1. Five Dysfunctions of a team

This page mainly exists as a tool for me to write up my key takeaways from Podcasts.

Best in breed procurement, where many systems - the best in class for each system - are procured independently then integrated with one another has had a tricky decade.

The underlying principle was good. Take lots of vendors who do just one thing really well and connect them to one another to form one fully integrated super system.

But multiple studies suggest that 70-85% of the integration projects which are essential to make these systems work together fail to achieve their objectives.

This post looks at what it’s necessary to consider to avoid these failures, especially in the workforce management space and breaks out specific things to explore when considering an integration project.

After 15 years of building technology businesses, I've changed my views on a lot of things from how to build software to what a good culture looks like.

A small subset of things have remained consistently true.

All of these are "true" for me, as always when I write "you" I really mean me. Writing is hard.

A fun thing about being alive now is how much knowledge and information is available in the form of books, blog posts, audio books, social media posts and YouTube videos to name just a few. But now more than ever there's no way we can consume all of it and so on which content we choose to focus our time and attention has the potential to drive wildly different outcomes.

In particular I believe an organisation that is driven heavily by short-form content loses it's ability both to reason about novel situations and discuss decisions constructively, leading to a lower likelihood of success for that organisation.

In the book "High Output Management," author Andrew Grove writes:

“Reports are more a medium of self-discipline than a way to communicate information. Writing the report is important; reading it often is not.”

Over the last ten years of building startups, this concept that the purpose of writing is often more to help the writer develop their thinking than it is to help others understand them has become increasingly central to the culture I aim to drive in an organisation.

A common topic of discussion is whether meetings should be used for information distribution or only discussion.

Probably the most famous example of a conclusion on this is the Amazon approach (documented in working backwards) where all meetings begin with some form of document and 15 minutes silence for people to read followed by discussion.

I'm a strong advocate of this approach but wanted to be better able to explain why I believe it's applicable to most situations and what's different about the small subset of exceptions.

So I attempted to break down the factors which play into it and what I believe is different between the possible approaches.

Deploying Rails to a VPS with Capistrano remains one of the simplest and most reliable methods for getting a Rails app up-and running. With the likes of Hetzner Cloud, Digital Ocean and Linode providing inexpensive, reliable virtual machines, Rails app serving substantial amounts of traffic can be hosted with minimal cost and complexity.

In the previous post we used Chef to prepare an Ubuntu 20.04 server for deployment of our Rails application. This included installing Nginx, PostgreSQL, Redis and our Ruby version of choice. We used Chef for this rather than entering command manually so that we can trivially create additional identical servers in future without needing to remember lots of terminal commands and config file edits.

In this tutorial we'll use Capistrano to automate deployment of our application, including generating all required config files, obtaining a free SSL certificate with Lets Encrypt and enabling zero downtime deployment.

Deploying Rails to a VPS with Capistrano remains one of the simplest and most reliable methods for getting a Rails app up-and running. With the likes of Hetzner Cloud, Digital Ocean and Linode providing inexpensive, reliable virtual machines, Rails app serving substantial amounts of traffic can be hosted with minimal cost and complexity.

We'll first use Chef to provision a VPS including securing and hardening the server, installing the correct Ruby version(s) and setting up Postgres and Redis. We'll then use Capistrano to deploy our Rails app, including appropriate systemd units to ensure our services are started automatically on boot .

Many guides to deploying Rails with Capistrano will use systemd to have it auto-started when the system boots. This is often done using the system instance of systemd which by default can only be controlled by root.

The typical workaround for this is either to grant our Capistrano deployment user passwordless sudo access or to grant them passwordless sudo access to just the commands required to restart the rails (and potentially sidekiq) systemd services.

This can be avoided by using the systemd user instance, which allows persistent services to be managed as a non-root user. This is compatible with the default systemd configuration in Ubuntu 20.04.

When using the capistrano puma gem with systemd, we may get the error:

Permission denied @ rb_io_reopen - /home/deploy/LOG_FILE_PATH/shared/log/puma_access.log (Errno::EACCES)

This may be caused by doubling up on the puma app servers logging.

Typically our systemd unit will contain something like:

StandardOutput=append:/home/deploy/LOG_FILE_PATH/shared/log/puma_access.log

Which means that any data written to standard output will be appended to the log file specified by systemd.

If we're getting the above error, it's also likely that our puma.rb configuration file contains something like:

stdout_redirect '/home/deploy/LOG_FILE_PATH/shared/log/puma_access.log', true

Which tells puma itself to write to a log file instead of to stdout.

This doubling up leads to the following:

• systemd creates the log file as the root user
• puma which we will generally have running as a different user then tries to write to this same file, but it doesn't have permission because it was created by root

The solution to this is simple, we can complete remove this line from puma.rb:

stdout_redirect '/home/deploy/LOG_FILE_PATH/shared/log/puma_access.log', true

Since the redirection of stdout is already being handled by systemd.

When trying to use the Capistrano Puma gem to restart Puma via systemd, we may run into an error along the lines of:

puma_APP_NAME.service is not active, cannot reload

This typically happens either because the service was never enabled or because in the time which elapsed between it being enabled and the first deploy taking place, it has crashed a sufficient number of times that it is no longer active.

The behaviour we want in this scenario is to reload the service if it is active, otherwise to restart it.

Happily systemctl [supports this out of the box]https://www.freedesktop.org/software/systemd/man/systemctl.html with systemctl reload-or-restart.

We can add the following to lib/capistrano/tasks to add a task which uses this to the puma namespace provided by the capistrano puma gem:

namespace :puma do
  namespace :systemd do
    desc 'Reload the puma service via systemd by sending USR1 (e.g. trigger a zero downtime deploy)'
    task :reload do
      on roles(fetch(:puma_role)) do
        if fetch(:puma_systemctl_user) == :system
          sudo "#{fetch(:puma_systemctl_bin)} reload-or-restart #{fetch(:puma_service_unit_name)}"
        else
          execute "#{fetch(:puma_systemctl_bin)}", "--user", "reload", fetch(:puma_service_unit_name)
          execute :loginctl, "enable-linger", fetch(:puma_lingering_user) if fetch(:puma_enable_lingering)
        end
      end
    end
  end
end

after 'deploy:finished', 'puma:systemd:reload'

This should be used in conjunction with including the puma systemd tasks in our Capfile using the load_hooks: false option which prevents the default restart task from being called.

install_plugin Capistrano::Puma::Systemd, load_hooks: false

The use of the above task also allows for zero downtime deploys when used with the relevant puma configuration and systemd unit file. See this post for more on the systemd unit file and this repository for a working example.

When attempting to deploy a Rails application using the puma web sever using the systemd functionality in the capistrano puma gem, we may receive the error message:

Neither a valid executable name nor an absolute path

When attempting to start the systemd service. This most often occurs when using the capistrano rbenv plugin. This is because the Capistrano rbenv plugin modifies the SSHKit.config.command_map[:bundle] path to include the RBENV_ROOT and RBENV_VERSION environment variables at the start of the bundle path. Systemd doesn't support Exec command starting with environment variables, instead requiring them to be in separate Environment lines.

We can fix this by overriding the puma.server.erb template with a new systemd unit file as follows:

[Unit]
Description=Puma HTTP Server for <%= "#{fetch(:application)} (#{fetch(:stage)})" %>
After=network.target

[Service]
Type=simple
<%="User=#{puma_user(@role)}" if fetch(:puma_systemctl_user) == :system %>
WorkingDirectory=<%= current_path %>
ExecStart=/usr/local/rbenv/bin/rbenv exec bundle exec puma -C <%= fetch(:puma_conf) %>
ExecReload=/bin/kill -USR1 $MAINPID
ExecStop=/bin/kill -TSTP $MAINPID
StandardOutput=append:<%= fetch(:puma_access_log) %>
StandardError=append:<%= fetch(:puma_error_log) %>
<%="EnvironmentFile=#{fetch(:puma_service_unit_env_file)}" if fetch(:puma_service_unit_env_file) %>
<% fetch(:puma_service_unit_env_vars, []).each do |environment_variable| %>
<%="Environment=#{environment_variable}" %>
<% end %>

Environment=RBENV_VERSION=<%= fetch(:rbenv_ruby) %>
Environment=RBENV_ROOT=/usr/local/rbenv

Restart=always
RestartSec=1

SyslogIdentifier=puma_<%= fetch(:application) %>_<%= fetch(:stage) %>

[Install]
WantedBy=<%=(fetch(:puma_systemctl_user) == :system) ? "multi-user.target" : "default.target"%>

Note that this hardcodes the path to rbenv so if the path is different, for example because it's a user install not a system install, this will need updating.

This unit file also adds an ExecReload option to allow us to use systemd for zero downtime deploys.

For a fully working example see this repository.

There's more information in this github issue.

In this series of posts we cover how to setup a comprehensive group based single sign on system for Kubernetes including the kubectl cli, any web application with ingress, a docker registry and gitea. We'll cover most of the common SSO models so adapting what's here to other applications such as Gitlab, Kibana, Grafana etc is simple.

The full solution uses Keycloak backed by OpenLDAP. OpenLDAP is required for the Gitea component, but can be skipped for the other components, including OIDC based SSO for kubectl.

Some of the highlights this series covers are:

1. Login to the kubectl cli using SSO credentials via the browser
2. Replace basic auth ingress annotations with equally simple but much more secure SSO annotations
3. Push and pull to a secure private docker registry with full ACL

1. Contents and overview
2. Installing OpenLDAP
3. Installing Keycloak
4. Linking Keycloak and OpenLDAP
5. OIDC Kubectl Login with Keycloak
6. Authenticate any web app using ingress annotations
7. Gitea (requires LDAP)
8. Simple Docker Registry
9. Harbor Docker Registry with ACL

Finally there were a lot of excellent resources I leant on when creating this series, there's a summary of the key ones here.

A commonly cited pain point for teams working with Kubernetes clusters is managing the configuration to connect to the cluster. All to often this ends up being either sending KUBECONFIG files with hardcoded credentials back and forth or fragile custom shell scripts wrapping the AWS or GCP cli's.

In this post we'll integrate Kubernetes with Keycloak so that when we execute a kubectl or helm command, if the user is not already authenticated, they'll be presented with a keycloak browser login where they can enter their credentials. No more sharing KUBECONFIG files and forgetting to export different KUBECONFIG paths!

We'll also configure group based access control, so we can, for example create a KubernetesAdminstrators group, and have all users in that group given cluster-admin access automatically.

When we remove a user from Keycloak (or remove them from the relevant groups within Keycloak) they will then lose access to the cluster (subject to token expiry).

For this we'll be using OpenID Connect, more here on how this works.

By default, configuring Kubernetes to support OIDC auth requires passing flags to the kubelet API server. The challenge with this approach is that only one such provider can be configured and managed Kubernetes offerings - e.g. GCP or AWS - use this for their proprietary IAM systems.

To address this we will use kube-oidc-proxy, a tool from Jetstack which allows us to connect to a proxy server which will manage OIDC authentication and use impersonation to give the authenticating user the required permissions. This approach has the benefit of being universal across clusters, so we don't have to follow different approaches for our managed vs unmanaged clusters.

This post is part of a series on single sign on for Kubernetes.