Fly.io Remote-Local Web development

I have recently been building a system with a web frontend that runs on Fly. Maintaining a local web development environment had been cumbersome and required maintaining mock APIs or running test versions of the APIs locally. With Fly's wireguard integration and some Openresty Lua scripting, I was able to avoid this trouble by serving my local web development server through staging and development environments on Fly.

I first tried to run a local test environment that would mock dependencies or run services in development mode. Here are just a few of the challenges I faced:

• General maintenance of dependency mocks and services
• Third party services and APIs may not be easily configured to run locally
• Local data may not be as rich or realistic as data in a staging environment
• Maintaining the environment for every developer's OS and configuration
• Subtle bugs emerge from the differences between local and cloud environments
• Domain configuration incompatibility with localhost and non TLS configurations

When I had to debug problems against a cloud enivornment, I would deploy to a development or staging environment in the cloud by rebuilding the assets, uploading them to the web server, and reloading the web page. I automated this process but it was still noticably slower than vite with HMR.

These frustrations led me to experiment with Fly's wireguard integration and some Openresty scripting to serve my local development web server through cloud development environments. With this setup, I aimed to have full parity between cloud environments and my development environment, while having the UX of a local environment with HMR.

Overview

The platform, deployed on Fly, uses Openresty to proxy requests to the API and serve web assets. Ordinarily, the proxy would serve existing web assets packaged in the docker image. Openresty can be configured to run custom Lua scripts at different phases of a request for each Nginx location block, which allows us in this situation to route requests upstream to developer's local servers through Fly's wireguard integration. We'll use a devid cookie set in the user's browser to lookup a user's local wireguard peer configuration in Redis that will then be used to construct an upstream domain to proxy. Although the diagram depicts the User and the development computer as separate, I'll walk through an example of this system with the user and development environment running on the same machine.

A. User setup and configuration

The first step is to create a wireguard interface to the Fly private network of your organization. Fly will by default create wireguard peers with the prefix interface-, which prevents them from appearing in Fly DNS records like _peer.internal TXT records or as <peername>._peer.internal records. We'll rely on <peername>._peer.internal domains to proxy requests from the Openresty proxy. To ensure wireguard peers are discoverable, create the wireguard peer manually using fly CLI.

fly wg create $YOUR_FLY_ORG yyz $WG_PEER_NAME

Finish the setup of the wireguard interface; on linux, this required adding the wireguard interface configuration to /etc/wireguard and initializing the interface with wg-quick. If you'd like to share your frontend dev server with a colleague, you can create a devid for them and submit your configuration to the proxy. They'll then have access to make requests to any local server bound to your wireguard interface.

The devid must be a cryptographically random token that can be generated by the user and set through the /webdev/set route. To prevent public access to this route, the Nginx location is configured to listen on a separate port that's not proxied by the Fly. For the sake of expedience I used the python REPL to generate my devid.

Here are two make tasks I used to help set my devid cookie in the browser. The first prints a JS snippet for the user to paste into their browser console to set the DEVID stored in their .env file. To set the cookie in the browser I use this simple snippet of JS in the browser's console document.cookie = 'devid=${DEVID}'.

devid_cookie_snippet:
	@echo "document.cookie = 'devid=${DEVID}'"

set_devid:
	@./scripts/setdevid.sh

The setdevid.sh script sets the local development environment configuration by making a POST request to the Openresty server through a wireguard connection to the organisation's Fly private network. The properties are saved by an Openresty Lua script to Redis with a key of devid mapped to the configuration data.

#! /usr/bin/env bash
set -e
# cd to the script directory
cd $(dirname $0)
# Get current public ip
MYIP=$(dig +short myip.opendns.com @resolver1.opendns.com); \
# Write dev info for webproxy
curl -X POST -d "devid=${DEVID}&peer_name=${PEER_NAME}&client_ip=${MYIP}&dev_port=5173" \
    http://${PROXY_SVC_NAME}.internal:8001/webdev/set

B. Request web assets

With the devid cookie set, a request to the platform can be made for the website at the / route. The cookie we set earlier will be available to the proxy to check against known web development configurations. Fly apps can be configured to serve from <APPNAME>.fly.dev or from a custom domain as described in the docs.

C. Request Handler

The two most important location blocks for this system are the / location and the /webdev/set location. Since other more specific locations are defined for /api and other routes, the / location is configured to receive requests for static assets. This location is configured to handle requests with a Lua script that checks for a devid cookie that maps to a data object stored in redis that contains these properties:

• client_ip - The expected source IP address of the request
• peer - The wireguard peer name
• dev_port - The port the local development server is running on

D. Retreive Default Web Assets

In the case that no devid cookie is present or no configuration is found, the default assets are served.

E. Proxy requests to the local environment

If a match is found in Redis, the source IP is compared to the client_ip and if they match then the domain name for the local machine is constructed using peer and dev_port. The domain format for wireguard peers, provided by Fly in their networks as described here, looks like this: <wg_peer_name>._peer.internal. In this example, the request is proxied to our development machine, and we should see a log of these requests in our local vite server. In the browser, changes made and saved in the editor will now be reflected in the browser through HMR, while requests to the API are routed to the API app in Fly.

Improvements

Here's a non-exhaustive list of improvemnets that can be made:

• Security improvements can be made. Generation and rotation of devid tokens can be automated and managedwith a Lua script in Openresty or with a dedicated service.
• At the moment, any developer with access to the organization's private network can set a devid and point it to a developers local wireguard peer. This should be limited to the owner of the wireguard peer.
• Alternatively, the devid can be replaced with OIDC integration and the proxy can be configured to search for a mapping between the user and a wireguard configuration.
• A dedicated service to manage the provisioning of devid tokens and storage of configuration