Ubuntu Mirror

Why do this?

I have many Ubuntu servers in my own data center at home. They need to update software packages every week.

However, if every server connects to the Ubuntu official source, it will waste a lot of bandwidth.

Therefore, I decided to set up an Ubuntu mirror on a server at home.

Step 1 - Get an Ubuntu server

Here, you can create a local server or purchase an Ubuntu server from a cloud service provider.

You can buy a pure server at Vultr, Azure, or DigitalOcean.

After you installing a pure Ubuntu, or buying a new cloud Ubuntu server, you need to first finish the basic authentication configuration.

Follow instructions here to:

• Delete other accounts.
• Create your own account instead of root.
• Disable password login and force to use SSH key to log in.
• Disable root account.
• Enable sudo without password.
• Enable firewall.
• Adjust timezone.
• Enable BBR

Best-practice for authentication after creating a new Linux server

Of course, you need to make sure that this Ubuntu server can access the Internet and can be accessed by your other servers.

Step 2 - Choose the source you want to mirror

Obviously, you can mirror the official source: http://archive.ubuntu.com/ubuntu/.

But if you are in China, you may find that the speed of the official source is very slow.

Therefore, you can choose some domestic sources, such as:

• http://mirrors.aliyun.com/ubuntu/
• http://mirrors.ustc.edu.cn/ubuntu/
• http://mirrors.tuna.tsinghua.edu.cn/ubuntu/
• http://mirrors.sohu.com/ubuntu/
• http://mirrors.163.com/ubuntu/
• http://mirrors.cloud.tencent.com/ubuntu/

These service providers generally have prominent pages that prompt you to use their sources, such as Tencent's https://mirrors.tencent.com/help/ubuntu.html.

It is recommended to test the speed and latency of each one using aria2c, ping, and wget.

Step 3 - Start Mirroring

First, you need to install a software called apt-mirror.

sudo apt-get install apt-mirror

You can use this project to patch the apt-mirror to fix possible c-n-f meta issue:

https://github.com/Stifler6996/apt-mirror

sudo wget https://github.com/Stifler6996/apt-mirror/archive/refs/heads/master.zip
sudo unzip master.zip
sudo mv ./apt-mirror-master/apt-mirror /usr/bin/apt-mirror
sudo rm -rf ./apt-mirror-master
sudo chown root:root /usr/bin/apt-mirror
sudo chmod 755 /usr/bin/apt-mirror

Then, you need to edit the /etc/apt/mirror.list file to specify the source you want to mirror.

sudo vim /etc/apt/mirror.list

Here, modify the mirror.list file to specify the source you want to mirror.

set base_path /var/spool/apt-mirror
set nthreads 20
set _tilde 0

deb http://mirrors.cloud.tencent.com/ubuntu/ jammy main restricted universe multiverse
deb http://mirrors.cloud.tencent.com/ubuntu/ jammy-updates main restricted universe multiverse
deb http://mirrors.cloud.tencent.com/ubuntu/ jammy-backports main restricted universe multiverse
deb http://mirrors.cloud.tencent.com/ubuntu/ jammy-security main restricted universe multiverse

clean http://mirrors.cloud.tencent.com/ubuntu/

In general, adding the above content is enough.

Where:

• main is the main software package
• restricted is the restricted software package
• universe is the community-maintained software package
• multiverse is the non-free software package
• jammy is the code name for Ubuntu 20.04. If you need other versions, you can refer to
  • jammy (22.04)
  • focal (20.04)
  • bionic (18.04)
  • xenial (16.04)
• focal-updates is the updated software package
• focal-backports is the backup software package
• focal-security is the secure software package
• clean is here to clean up some unnecessary software packages

After the modification is completed, simply run the apt-mirror command to start Mirroring.

After the Mirroring is completed, you can see the files you have downloaded. They will be placed in the /var/spool/apt-mirror directory.

These files may be several hundred GB, so you need a relatively large hard drive. This process may take several hours.

sudo apt-mirror

Step 4 - (Optional) Mirror ppa

As we all known, http://ppa.launchpad.net/ is extreamly slow. You can also mirror it.

For example, I'm mirroring Nextcloud client and Firefox. Simply add this to your /etc/apt/mirror.list:

deb http://ppa.launchpad.net/nextcloud-devs/client/ubuntu jammy main
deb http://ppa.launchpad.net/mozillateam/ppa/ubuntu jammy main

If you want to mirror both AMD64 and i386 versions:

deb-amd64 http://ppa.launchpad.net/nextcloud-devs/client/ubuntu jammy main
deb-amd64 http://ppa.launchpad.net/mozillateam/ppa/ubuntu jammy main
deb-i386 http://ppa.launchpad.net/nextcloud-devs/client/ubuntu jammy main
deb-i386 http://ppa.launchpad.net/mozillateam/ppa/ubuntu jammy main

Now run sudo apt-mirror again!

Step 5 - Configure your web server

Obviously, in order for your other servers to be able to access these software packages, you need a web server.

Here, I use Aiursoft.Static to create a simple web server.

sudo apt install dotnet8
sudo dotnet tool install Aiursoft.Static --tool-path /opt/static || sudo dotnet tool update Aiursoft.Static --tool-path /opt/static

This will install Aiursoft.Static in the /opt/static directory.

At the same time, the above script can also update Aiursoft.Static.

In order for the /var/spool/apt-mirror directory to be accessible by Aiursoft.Static, you need to modify the permissions.

sudo chown -R apt-mirror:apt-mirror /var/spool/apt-mirror
find /var/spool/apt-mirror/ -type d -print0 | sudo xargs -0 chmod 0755
find /var/spool/apt-mirror/ -type f -print0 | sudo xargs -0 chmod 0644

This setting is to allow the apt-mirror user to read and write to this directory, while other users, such as the www-data user, can only read it. This way, your web server can run completely under the www-data user.

Next, create a systemd service to automatically run Aiursoft.Static when the system starts up.

sudo vim /etc/systemd/system/mirror-web.service

Here, edit the apt-mirror.service file and write the following content:

[Unit]
Description=Apt Mirror Web Server
After=network.target
Wants=network.target

# Before starting, run:
# sudo chown -R apt-mirror:apt-mirror /var/spool/apt-mirror
# find /var/spool/apt-mirror/ -type d -print0 | sudo xargs -0 chmod 0755
# find /var/spool/apt-mirror/ -type f -print0 | sudo xargs -0 chmod 0644

# So www-data user can read the files
# Also apt-mirror user can write to the files
[Service]
User=www-data
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/opt/static/static --path /var/spool/apt-mirror/mirror/mirrors.cloud.tencent.com/ -p 12386 --allow-directory-browsing
WorkingDirectory=/var/spool/apt-mirror/mirror/mirrors.cloud.tencent.com/
LimitNOFILE=1048576
KillSignal=SIGINT
Environment="ASPNETCORE_ENVIRONMENT=Production"
Environment="DOTNET_PRINT_TELEMETRY_MESSAGE=false"
Environment="DOTNET_CLI_TELEMETRY_OPTOUT=1"
Environment="ASPNETCORE_FORWARDEDHEADERS_ENABLED=true"

[Install]
WantedBy=multi-user.target

The purpose of this file is to automatically run Aiursoft.Static when your system starts up.

Then, start this service.

sudo systemctl enable mirror-web
sudo systemctl start mirror-web
sudo systemctl status mirror-web

Now you can try to browse http://<your server's IP address>:12386 to see if you can access your software package.

Step 6 - Configure reverse proxy

Obviously, exposing port 12386 directly on the public network seems very strange. Generally, I like to use Caddy to do a reverse proxy.

http://<your domain> http://<your domain> {
    reverse_proxy / http://<your server's IP address>:12386
}

This way, you can access your software package through http://<your domain>. Of course, if your client supports HTTPS, it will also use HTTPS automatically.

apt has always liked to hardcode http, so the above configuration can explicitly tell caddy to turn off https redirection.

Step 7 - Configure your client

Finally, you need to configure your client so that they can use your mirror source.

Here, you need to edit the /etc/apt/sources.list file to specify your mirror source.

sudo vim /etc/apt/sources.list

Simply change its contents to your own server!

deb http://<your domain>/ubuntu/ jammy main restricted universe multiverse
deb http://<your domain>/ubuntu/ jammy-updates main restricted universe multiverse
deb http://<your domain>/ubuntu/ jammy-backports main restricted universe multiverse
deb http://<your domain>/ubuntu/ jammy-security main restricted universe multiverse

Okay, try sudo apt update!

Of course, you may want to clean up the cache, you can use sudo apt clean.

Conclusion

Well, this is the process of setting up an Ubuntu mirror. It can help you save a lot of bandwidth and speed up the download of your software packages.

You can also share the client configuration method with your friends so that they can also use your mirror source.