Usage

Adjust any repos pointing to mirrorlist.centos.org to centosmirrorlist.randu.me.  

sed -i 's/http://mirrorlist.centos.org/http://centosmirrorlist.randu.me/g' /etc/yum.repos.d/*.repo

Architecture

Some of the choices here were a combination of using what I already had available and trying some tools I have not previously used.

The code backing this is a very simple Go application using no external libraries and is around ~100 LOC.

I set up a new service using Google Cloud Run using the console.  During setup, this gave an option to automatically pull from github and integrate their CI/CD system.  Any push to the main branch of the repo will automatically build and deploy.

In front of all this is a pull endpoint on bunny cdn serving out the updated mirror list to the public.  The CDN endpoint is configured to serve cached responses during update and if the source service is offline.

Server List

At the moment, the following vault sources are returned:

• vault.centos.org
• archive.kernel.org/centos-vault
• mirror.nsc.liu.se/centos-store

The application will make it easy to add/remove mirrors, including private mirrors as a safety mechanism in the event that IBM decides to push an update killing the vault or in the event that someone puts up an updates mirror containing packages from extended support RHEL 7.

Detect a bot by missing name

Most of the web has gone HTTPS.  Security certificates are usually for a single hostname or a wildcard within a domain.  The old practice of having a fallback to a default host can be turned on its head here.  Set up you virtualhosts as usual, but set the fallback virtualhost to a static page explaining the user has been blocked.  Use your server's log formatting to mimic a bot attack that would be caught by your normal bot detection systems.

Detect a bot by HTTP protocol version

This one leaves open the possibility of false-positives if you have things other than web browsers hitting your site.

Most libraries used by bots and various programming environments use HTTP 1.1.  Any connection not using IE, but using HTTP/1.1 for TLS is unlikely to be a real user.

Detect a bot by HTTP status code

I don't think I've ever seen a HTTP 400 result from a legitimate request.

One odd outlier is playback of raw pcm data or WAV files.  Recent versions of Chrome, Edge and Firefox all play back WAV files with an audio tag without any problem.  Internet explorer will not.  MSE also does not specify an option for PCM or WAV playback.  MSE also requires IE 11.  The Web Audio APIs / AudioBuffer allow this, but they are not available in any version of IE.

I thought about the problem a bit and remembered the <bgsound> tag that has existed since early versions of IE... and confirmed that it will indeed play WAV files.  The only problem is, there is no way of managing controls / seeking.  I put together a proof of concept to work around this limitation.

The WAV file format is easy to parse.  My implementation here is taking quite a few shortcuts... so it might not work with all files.  The solution is pretty simple... read the WAV file as an ArrayBuffer with XHR, seek by modifying the size fields in the header and cutting the output... then feed a Blob as a URL for the src attribute to <bgsound>.  If you were working with larger files, you could use 2 range requests – 1 for the header and 1 for the body.  My method below works under the assumption that seek performance will be improved by the browser cache.

function playWavIE(url, seek, callback) {
	xhr = new XMLHttpRequest();
	xhr.onload = function(e) {
		var view = new DataView(xhr.response);
		var fmt = {
			fileSize: view.getUint32(4,true),
			dataType: view.getUint16(20,true),
			channels: view.getUint16(22,true),
			sampleRate: view.getUint32(24,true),
			sampleValue: view.getUint32(28,true),
			sampleBytes: view.getUint16(32,true),
			bitsPerSample: view.getUint16(34,true),
			dataSize: view.getUint32(40,true)
		};
		fmt.duration = fmt.dataSize / fmt.sampleValue;

		if(seek> 0) {
			var cutBytes =  Math.floor(fmt.sampleValue*seek);
			var header = xhr.response.slice(0, 44);
			var hview = new DataView(header);
			hview.setUint32(4, fmt.fileSize - cutBytes,true);
			hview.setUint32(40, fmt.dataSize - cutBytes,true);

			blob = new Blob([header, xhr.response.slice(hview.byteLength + cutBytes)], {"type": "audio/x-wav"});
			url = URL.createObjectURL(blob);
		}

		if(document.getElementsByTagName("bgsound").length) {
			var bgsound = document.getElementsByTagName("bgsound")[0]
		}
		else {
			bgsound = document.createElement("bgsound");
			document.body.appendChild(bgsound);
		}

		bgsound.src = url;
		if(callback !== undefined) {
			callback(fmt);
		}

	}
	xhr.open('GET', url, true);
	xhr.responseType = 'arraybuffer';
	xhr.send();
}

function playCB(fmt) {
	console.log("playCB called");
	console.log(fmt);
}

My first draft produced code that could have a number of problems... the biggest being security.  As it imported the python file, code could be executed.  Another likely problem revolves around long running processes and how python handles imports.  Even if the import is handled in a function's namespace, the setup functions, etc will only be executed once.  Even trying to catch for this with re-import, the way the internal dictionaries are updated would likely result in unexpected behavior.

This may not be the ideal solution, but it is fairly safe (no code execution) and still uses the CPython interpreter/compiler rather than trying to parse with regex, etc.  This is a quick bit of code and does not include proper error handling.

#!/bin/env python3
# NOTE: requires python >= 3.2

import parser

def moduleFunctions(fileName):
	d = {}

	with open(fileName, 'r') as myfile:
		code_str = myfile.read()

	code_obj = parser.suite(code_str).compile()
	for obj in code_obj.co_consts:
		if isinstance(obj, type(code_obj)):
			d[obj.co_name] = obj.co_varnames[slice(obj.co_argcount)]
	return d

SSH & iptables geo restriction

SSH brute force attacks are incredibly common. I primarily use sshguard toauto-block individual IPs after a certain number of failed requests. This willcontinue, but I wanted to add another layer to reduce the number of connectionsmaking it to my SSH server processes without either changing the…

randuBrad Murray

At times, you may want to restrict (or open) access to a particular network that may have many IP ranges.  The most common use cases would be to whitelist access from a corporate network or a specific ISP.

If you don't know the ASN you will be working with, I suggest using bgp.he.net to look it up.

Here is a quick bash script to create an ipset for a particular ASN.

#!/bin/bash
if [ ! -x "/usr/bin/whois" ]; then
        echo "whois must be installed at /usr/bin/whois"
        exit 2
elif [ $# -ne 1 ]; then
        echo "Usage $0 [ASN #]"
        exit 1
elif ! [ $1 -eq $1 ] 2> /dev/null; then
        echo "ASN must be a number"
        exit 1
fi

setname="as$1"

echo "create $setname hash:net family inet hashsize 1024 maxelem 2048" > "$setname.ipset"
/usr/bin/whois -h whois.radb.net -i origin AS$1 | /bin/grep -Eo '([0-9.]+){4}/[0-9]+' | sed "s/^/add $setname /" >> "$setname.ipset"
lines=`wc -l < $setname.ipset`

if (($lines < 2)); then
        echo "ASN $1 not found or no routes found"
        rm -f "$setname.ipset"
        exit
elif (($lines > 2049)); then
        echo "More than 2048 records found, adjust maxelem for set"
fi

A Recap

Among other things, we use terraform to scale CDN edge capacity, as well as custom compute instances for live video transcoding.

We use multiple cloud providers with overlapping locations and preferred data centers for each location.  The preference is based on a combination of factors including transit providers at the location, long term reliability and cost.  In most cases we use Linode, Vultr or Hetzner, although we are configured to work with additional providers.

Linode is generally our top choice, primarily because our data transfer is high and they pool all allocated transfer.  Linode's host instances are on 40G links, all VMs include a fair amount of data transfer (per hour/month) and the per-VM port speeds are also generous.

We do have issues with Linode, as CPU steal is sometimes high.  This is mostly an issue with their standard plans.  In the past, I've also been disappointed with the N Cal data center, due to both power problems and transit (at the time, mostly HE).

We have seen times when steal bumps up a bit, even on their dedicated CPU instances.  I suspect these are sold by the thread, not by the core so on very rare occasion you may end up on a host that is heavily loaded.

Commands to start and stop instances can be triggered by our clients, by our backend systems or through scheduling.  All of this is implemented through our own scripts.

Troubleshooting and Improvements

When we first started offering services that included live transcoding, we occasionally got complaints of video breakup, dropped frames, etc.  It didn't take long to confirm that this was caused by steal (oversold cpu).  Buying a larger plan doesn't work, as a single low performance core breaks things.  For most applications, slight inconsistencies in performance aren't a huge problem.  In the case of video transcoding, it makes a huge difference.

When we first started doing this, Linode didn't offer 'Dedicated CPU' plans, either.

The first step was a simple warning.  I added a method in the provisioning scripts to run mpstat and send an alert to my devices through pushover if the steal rate at time of provisioning was too high (>1%).  That 1% steal has virtually no effect on a web server, but these aren't web servers.

The second step was returning an error code on the provisioning script if the steal was too high.  To prevent infinite loops (and large bills) I added a safeguard in our provisioning script that keeps a count of retries (that resets after a full success). The script also parses the output to determine which VM is failing.  In our case, there are often multiple VMs provisioning at any time.  After too many retries, it sends a high priority alert via pushover and prevents future attempts at running until a manual reset.  This made a huge difference in the amount of manual effort involved, but there were still occasions where after a few retries, it could not get a host with a clean CPU.

Failover

The final solution builds on the above, but adds a failover between plans and providers.  Our generated .tf files now add a comment line with json encoded data containing important metadata such as the provider used, datacenter, cpu/plan type and retry count.  Our wrapper parses the output from terraform apply and keeps track of which VMs failed due to CPU steal.  After terraform completes its operation, this information is used to run a failover function.  

We have an internal flow used to determine a failover plan.  Here's an example...

Initial configuration was for a standard 6 core linode instance.  On the first try, it spots CPU steal.  Regenerate the configuration, but with an 8 core dedicated (there is no 6 core standard).  There is still a failure (this is very rare), regenerate the configuration at Vultr.  There is still a failure (this hasn't happened in real world only in our simulated environment), try a different Vultr data center.

Go for it.  You'll miss out on many opportunities by being too cautious.  Fear and failure are part of life.  Learn from the failures and you'll learn to tolerate the fear.

You're right that being different can be a very good thing... but learning to relate is important.

Get an outside perspective.  Learn to respect the opinion of others even when they are misinformed or under-informed.  Get perspective from other cultures.  Even if you reject most of their ideas, you gain understanding and the ability to think flexibly.  You might even change your mind on important matters.

Yes, substance is more important than style -- but in a marketplace of ideas, style often wins -- and as a result has more influence.  Always take this into consideration, and you won't be so surprised at the way things go.

Most people have no problem with initiating violence to accomplish their goals or allay their fears.  This force is usually carried out through government, but not always.  Most of these people don't even recognize their actions are violent and will continue to deny it even when it's made clear.

There's nothing wrong with being wrong.  It's only foolish when you choose to stay that way after being confronted with the facts.

Don't fault people for valuing feelings over facts.  Although the heart is often wrong, validated facts can do just as much damage when applied without feeling.  (for an extreme example, think German bioethics).

Geographic Solutions

I found a number of people were using maxmind's geolite country database with tcp wrappers / hosts.allow and a shell script.  I really don't want a shell process spawned with every request, I want them blocked by the firewall.  I looked at the plain text dump of this database, and was surprised to see a number of /32 addresses in a country level database.  I didn't want a crazy number of entries added to iptables.

I then looked at using the simple IANA assignments of /8 addresses.  This would probably work for me... as to my knowledge there aren't any /8's assigned to outside regions that contain smaller assignments back to ARIN.

After a little more digging, I found a site/service that offers regularly updated aggregated IP blocks by country.  The sizes here are reasonable, with around 18,000 blocks for the US.

netfilter sets

After a little digging and performance tests, it looks like creating whitelists will work fine.  I'm not a fan of firewalld, so I will use iptables/ipset for this.

Note: All sets must be created before the iptables rules are generated.  It's OK if they are empty, with the notable exception of the whitelist, as you don't want to risk getting locked out of your server.

To prepare the sets, I created a quick bash script.  I found something similar written by someone else, but it was very slow for large numbers of networks, as it spawned a process for every row being added.

#!/bin/bash
if [ $# -ne 1 ];
	then echo "Usage $0 [2 char country code]"
	exit 1
fi

url="http://www.ipdeny.com/ipblocks/data/aggregated/$1-aggregated.zone"
setname="geo_$1"

echo "create $setname hash:net family inet hashsize 8192 maxelem 65536" > "$setname.ipset"
curl -s $url | sed "s/^/add $setname /" >> "$setname.ipset"

This will create a file that can be imported using: cat $filename | ipset restore

iptables rules

We use sshguard to handle temporary blocking for those doing brute-force attacks.

For SSH:

• Allow IPs in the whitelist
• Block anything in the sshguard blocklist
• Allow anything else in the US (or your country/countries of choice)
• Block anything in the rest of the world

*filter

# Allow all loopback (lo0) traffic and reject traffic
# to localhost that does not originate from lo0.
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -s 127.0.0.0/8 -j REJECT

# Allow ping.
-A INPUT -p icmp -m state --state NEW --icmp-type 8 -j ACCEPT

# Allow mosh
-A INPUT -p udp -m multiport --dports 60000:61000 -j ACCEPT

# Allow SSH connections from whitelist
-A INPUT -p tcp --dport 22 -m state --state NEW -m set --match-set whitelist src -j ACCEPT

# Block SSH from sshguard blocklist
-A INPUT -p tcp --dport 22 -m state --state NEW -m set --match-set sshguard4 src -j REJECT

# Allow SSH connections from usa
-A INPUT -p tcp --dport 22 -m state --state NEW -m set --match-set geo_us src -j ACCEPT

# Allow HTTP and HTTPS connections from anywhere
# Allow RTMP from anywhere
-A INPUT -p tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -p tcp --dport 443 -m state --state NEW -j ACCEPT
-A INPUT -p tcp --dport 1935 -m state --state NEW -j ACCEPT

# Allow inbound traffic from established connections.
# This includes ICMP error returns.
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Reject all other inbound.
-A INPUT -j REJECT

# Reject all traffic forwarding.
-A FORWARD -j REJECT

COMMIT

Making it work on CentOS (7/8)

CentOS 7+ by default uses firewalld.  It must be disabled and iptables/ipset tools and systemd unit files must be installed

yum -y install iptables iptables-services ipset ipset-service
systemctl disable firewalld
systemctl stop firewalld
systemctl mask firewalld

systemctl enable iptables
systemctl enable ipset

After you have finished creating your rules and sets, you'll want to save them.

/usr/libexec/ipset/ipset.start-stop save whitelist
/usr/libexec/ipset/ipset.start-stop save sshguard4
/usr/libexec/ipset/ipset.start-stop save geo_us
/usr/libexec/iptables-iptables.init save

The b2 api is reasonably nice, although it has 1 major problem – tokens for small file uploads are not secure in a way that would allow them to be passed back to the client.

I started with resumable.js.  We've been using a modified version of this to handle large media uploads for a few years now.  It's a simple, relatively clean bit of code without any major external dependencies.  We had to make a few changes to resumable.js to allow our extension to work.

The frontend is a set of callbacks for resumable.js to perform sha1 hashing, fetch tokens, alter upload addresses, etc.

The backend (written in PHP 7) handles authenticating end user upload requests and creating backblaze tokens.  It also handles receiving small (up to 5MB) files from the end user and uploading them to b2.  The only thing it uses that may not be in everyone's PHP stack is redis, which is optional but highly recommended for performance.

A work in progress...

• More testing
• Example page with upload progress reporting
• Better handling of error conditions / reporting back to resumable.js
• Multi-threaded uploads of large files (we do this on our previous custom version of resumable.js)

I've been reviewing options for Object Storage.  For many years, we've maintained our own systems for this purpose.  Right now, most of our data is stored on a set of colocated storage servers (in different data centers in the same city) running zfs in a raidz2 configuration.  Data is mirrored between servers using lsyncd.

There are many factors to consider, although most stick with the option given by their primary cloud or datacenter provider.

Features and Pricing Overview

BackBlaze B2

BackBlaze have been offering cloud backup since 2007 and b2 object storage since 2015.  They have a long history of providing a quality product at a low price.

Their strength lies in their solid track record and low straight-forward pricing.  The biggest weakness is the datacenter options.  In the US, your data will be stored in either Sacramento CA or Phoenix AZ.  You do not have a choice of data centers.  In Europe, your data will be stored in Amsterdam.

The b2 protocol is fairly easy to work with.  One gotcha – there is no way to pass a token to allow upload of a specific file smaller than 5MB.  This isn't a problem for backups or transfers from servers you control.  It is a problem if you want to allow your end users to upload directly to B2.

The recent addition of the S3 protocol for new buckets, makes it much easier to integrate into existing code bases without vendor lock-in.  IAM-like features are much more limited than in AWS or Wasabi.

Wasabi

Wasabi is a newer offering (launched May 2017) started by the co-founders of Carbonite (online backup, launched in 2006).  It's well funded and likely to be a strong contender for many years.

Since Wasabi uses the S3 protocol and supports many of the same IAM features as Amazon's offering, tool/code options are good and migration should be reasonably simple for those currently using S3.

They offer storage in US-East (Ashburn VA and Manassas VA), US-West (Hillsboro OR), Europe (Amsterdam NL) and Asia (Tokyo JP) and allow you to select the data center for each bucket.

One thing that makes Wasabi unique is the elimination of transfer and API call fees.  This model only works if your average monthly transfer is less than your storage.  It also only works if you have a limited amount of transient data, as you will be billed for 90 days for every file stored.  It is my understanding that if you exceed the transfer limits, you will be asked to move to their older pricing model ($3.90/TB storage + $40/TB transfer).

Linode

Linode's Object Storage is now available in Newark, Frankfurt and Singapore.  The minimum price is $5/mo and includes 250GB storage and 1TB t ransfer.  Additional storage is $0.02/GB and outbound transfer is $0.01/GB.  I suspect it will mostly be attractive to those who have large deployments using Linode's computing systems.  The biggest benefits will likely be fast internal network traffic to your Linode instances and using your existing transfer pool for data transfer, reducing costs.

Vultr

Vultr's Object storage is a new offering.  Availability is currently limited to Newark / US-East.

Amazon S3

Amazon's S3 service is the big contender, although the pricing model is very complex and can get very expensive.  The biggest advantage of S3 is the tight integration with Amazon's other services.  When configured properly, data transfer from S3 to other AWS services is free... although API call fees can be significant.

Google Object Storage

Google object storage, similar to S3 would mostly be useful to services inside GCP.

For on-demand video, Roku devices use a custom file format that stores a number of images (video frames) in a single file.  This file is used for trick-play where you can see a preview of what the video looks like when you fastforward and rewind.

The format is a very simple archive format with an index and a number of JPGs embedded.  We re-use the Roku BIF files with a custom clappr player plugin (we will likely release this as well) along with a mouseover effect on our video lists page.

github page with source code

Example using our library

The armbian images are shippped as 7z compressed disk images.  This may be board specific, but the image I used contained a MBR boot record as well as a single ext4 partition.  This partition is resized on first boot.

To modify the image, I have a debian development environment running under VirtualBox.

losetup -P /dev/loop0 filename.img (this sets up the loopback device and forces a partition scan)

All my configuration is done from a script called from rc.local.  It changes permission to 000 on rc.local when it is complete.  Among other things, this installs additional packages I include in the distribution.

From another identical SBC with a clean install of Armbian, I run the following to fetch the necessary packages

apt-get clean && apt-get install --download-only packge1 package2 pacakge3

where package1, etc are all the packages I want to add.  The deb files and necessary dependencies will be at /var/cache/apt/archives.  These can be copied onto the image and installation done by the first boot script.

After unmounting the loopback device, it can be written back to an SD card using dd on Linux or balenaEthcer, etc on Windows.

dd if=armbianimage.img of=/dev/sdc bs=1M oflag=direct status=progress

Consistency and Availability

There is no cloud, it's just someone else's computer

One of the things I've spent quite a bit of time dealing with is keeping consistent performance.  How important this is depends on your usage scenario, but in our case (streaming media) it's vital.  Be aware that most of these problems are not the rule, but rather the exception.  Other than hyper-budget providers, most VPS/cloud providers manage resources well.... but when they do happen, they could ruin your day.

We take a hybrid approach to deployments – a combination of colo, dedicated servers, vps/cloud computing and SaaS using a variety of providers.  Our colo systems are mirrored to multiple providers, and we try to keep anything on a VPS/CCI set up for automated deployment... but not all instances are created equal – even when on the same plan with the same provider.

CPU Steal

CPU steal can be caused by a number of factors... but it's effectively a noisy neighbor problem (Linode) or throttling (AWS t2/lightsail).  Since multiple clients are using the same computer/host node, your processes may slow down significantly due to what other clients are doing.  In my experience, both networking and encoding performance are significantly effected with a CPU steal value higher than 2%.  Our provisioning scripts check the kernel steal value and exit / report a failed installation with a value higher than 1%.

Availability

On rare occasion, cloud provisioning systems become unavailable.  This is why we aren't dependent on a single provider.

We can easily switch a task to a different provider or data center in about 3 minutes.

We have also seen times where larger VMs are temporarily unavailable at a particular data center (mostly happens with Vultr).  For this case, we run a preflight check before generating provisioning commands for terraform confirming availability.  If the provisioning still fails, we switch the VM to a different data center.

Some provider details...

This covers only some of the cloud providers we use.  Later, I may cover my experiences with dedicated and colo providers.

Linode

Performance here is usually good, but we have to monitor things closely. Linode hosts nodes are all connected at 40gbps, and plans offer generous bandwidth caps, which is fantastic for our purposes, and we seldom see any real network issues.  Linode also pools transfer quotas across instances, which is also great for our relatively high volume, high burst usage scenario.  We have, however seen problems with CPU congestion/overselling, though never ran into throttling.  Here are some of the processors we have encountered with Linode.  Note that I count threads here rather than cores, as all providers I have encountered have HT enabled and assign CPU based on threads rather than running without HT and locking CPU affinity to specific cores.

Vultr

CPU performance has been more consistent with vultr.  I believe each host is connected at 10gbps.  To the best of my knowledge, vultr does not publish available port speeds/caps.... although 1gbps is pretty safe.  Since VPS nodes support AVX512 and I haven't had a problem with CPU steal, so far they have worked well for transcoder instances.

Vultr also offers 'bare metal' instances which are great all around at the current price point.  They connect at 10gbps and are dedicated, so no concerns about noisy neighbors.  Bare metal instances take a little longer to provision... typically around 5 minutes (while VPS provisioning at most providers is under 2 minutes).

Amazon

We mostly use amazon for AI services.  We seldom use their cloud computing products.  The price points for EC2 are not competitive with other options, especially for high bandwidth consumption.  Lightsail is priced appropriately but has poor performance (speculated that these are T2 burstable EC2 under the hood).  Both CPU and network are heavily throttled after about 20-30 minutes.  This can be detected both with benchmark software and kernel CPU steal information.

Digital Ocean

We seldom use Digital Ocean, mostly due to network performance.  The listed port speed is 1gbps, but performance is inconsistent, likely due to other clients using the same host name.  We keep them configured as another backup provider for transcoding.

Hetzner Cloud

Hetzner's offering is very nice and the price is fantastic, although data centers are only in Germany and Finland.  We mostly use Hetzner for overflow for on demand video encoding.  I believe each host is connected at 10gbps.  CPU is Xeon Scalable (Skylake-SP) @ 2.1ghz.  I am unsure on the specific model.  Performance is great for encoding, since Skylake supports AVX-512.