Stress testing RTMP server

After working with Nginx-RTMP, I wanted to see how much traffic I could actually put through my server setup. I tried just streaming a bunch of video feeds from the server, but I ran out of processing power on my desktop. I had to figure out a better solution.

Enter Flazr. Flazr is a Java application that allows you to, among other things, stress test an RTMP server.

What you’ll need is:

  • RTMP target – Hopefully with a FAT network connection
  • RTMP Client – This should be a device watching a video feed from the server, making sure video problems don’t pop up
  • RTMP Stress tester – This is the machine running Flazr. I used an Atom computer running Ubuntu 14.04 server for my Flazr machine. It should run on anything that runs Java.

For basic usage, just use the command line to initiate the test. Here’s an example of a configuration I used.

./client.sh -load 100 rtmp://10.0.0.10/live/willstare

The stream I was pulling was at 1500kbps. Here’s the bandwidth being used on the RTMP server.

 rx: 6.20 Mbit/s 8944 p/s tx: 169.46 Mbit/s 18667 p/s

And here’s the CPU usage on the RTMP server.

 load average: 0.01, 0.02, 0.05

Obviously, my server can handle more than 100 connections. Let’s crank it up to 675! 675*1.5mbps = 975mbps.. That’s just about hitting the top end of gigabit.

Before we do that, we should probably increase the heap size for Flazr. On Ubuntu, type nano cleint.sh to edit the script. Change -Xmx512m to -Xmx2048m (if you have at least 4gb of RAM in the machine).

./client.sh -load 675 rtmp://10.0.0.10/live/willstare

Server bandwidth @ 675 clients

 rx: 6.04 Mbit/s 8655 p/s tx: 961.13 Mbit/s 81735 p/s

And server load @ 675 clients

load average: 0.07, 0.03, 0.05

As you can see, RTMP streaming is very efficient. It only requires massive bandwidth. Even with 675 clients, the video was still smooth when viewing it from another machine.

Stress testing Shoutcast Server

If you’re like me, you’ve got a shoutcast server configured to handle thousands of connections. I was pretty sure my setup could handle the load, but I wanted to actually find out. After looking around online, I found out an easy way to stress test the server using curl. Here’s a little script I wrote that will allow you to murder your server :D. Run this script on a test machine with a decent internet connection so you can really stress the server. This script should run on basically any linux operating system. I used Ubuntu as my test box.

Just a few things before you run the script.. You will need vnstat (or you’ll have to comment it out) to get the most of the script. Also, be careful with how many threads you run. If your server has the ulimit too low, you could lock up the shoutcast server.

#!/bin/bash
echo "enter a URL to stress test. ex: http://radio.derp.com:8123"
read URL
echo "enter number of threads to run. ex: 50"
read THREADS

for ((N=0; N<$THREADS; N++))
	do  curl -o /dev/null $URL >> /dev/null 2>&1 &
done
echo "Created "$THREADS" threads connected to "$URL
echo "Live Bandwidth"
echo "hit ctrl-c when you are finished" 
vnstat -l 
echo "Press any button and hit enter to kill all curl processes! Thanks!"
read Q
killall -e curl

If you have any questions please leave a comment below. Thanks

Here’s an example of the script in action.

./shoutcast_stress.sh 
enter a URL to stress test. ex: http://radio.derp.com:8123
stream.willstare.com:8123
enter number of threads to run. ex: 50
1500
Created 1500 threads connected to stream.willstare.com:8123
Live Bandwidth
hit ctrl-c when you are finished
Monitoring em0... (press CTRL-C to stop)

 rx: 50.45 Mbit/s 4514 p/s tx: 2.48 Mbit/s 4503 p/s

And bandwidth usage on the server

 rx: 4.14 Mbit/s 7706 p/s tx: 66.88 Mbit/s 8105 p/s

And what Shoutcast looks like with 1500+ connections:

shoutcast stress

There’s a few glitches (backspace might now work when entering stream URL), but I think its a good tool.

Nginx-RTMP & WordPress Debian

Alright, let’s get started. This guide assumes you meet the following prerequisites:

  • Fresh Debain 7 64 bit installation (no ‘web server’) or any other packages installed.
  • 4 CPU, 1G memory, 250+gb HDD
  • SSH, sudo, NTP, and a static IP address configured
  • User account streamer with sudo
  • DNS SERVER – It is highly recommended you have a local DNS server to handle requests to your server. This will allow the hard-coded URLs in wordpress and the script to continue working even if the IP addressing scheme changes.

Everything can be done to a physical installation or a virtual machine.

Install NGINX-RTMP

First we will install nginx-rtmp from a PPA. These steps are taken from HERE.

sudo apt-get install dpkg-dev
sudo mkdir /usr/src/nginx
cd /usr/src/nginx
sudo apt-get source nginx
sudo apt-get install git-core
sudo git clone https://github.com/arut/nginx-rtmp-module.git
cd nginx-1.2.1/

 

– Note: This will depend on what version you have. As of this writing, nginx was at 1.2.1.

 

List the directory. The following files should be shown in /usr/src/nginx/nginx-1.2.1/

sudo nano debian/rules

 

Add the following to the add-modules configuration.

 module=/usr/src/nginx/nginx-rtmp-module \

 

So it looks like this

--add-module=$(MODULESDIR)/nginx-upstream-fair \
--add-module=$(MODULESDIR)/nginx-dav-ext-module \
--add-module=/usr/src/nginx/nginx-rtmp-module \ 

 

sudo apt-get build-dep nginx
sudo dpkg-buildpackage -b

 

Building the package may take a while.

cd ..
sudo --install nginx-common_1.2.1-2.2+wheezy3_all.deb nginx-full_1.2.1-2.2+wheezy3_amd64.deb
sudo service nginx start

 

Great! Now we have nginx-rtmp installed on our Debian installation. Next, we will configure some stuff within Nginx!

NGINX-RTMP Configuration for WordPress AND Streaming

Start by changing nginx worker processes to 1. This is to make sure the streaming stats don’t get all screwy.

sudo nano /etc/nginx/nginx.conf

So it looks like this.

user www-data;
worker_processes 1;
pid /var/run/nginx.pid;

 

Then scroll down and add the following to the end of the file.

rtmp {
        server {
                listen 1935;
                chunk_size 4096;

                application live {
                        live on;
                        record on;
                        record_path /opt/nginx_scripts/live_recording/;
                        record_suffix -%d-%b-%y-%T.flv;
                        record_lock on;
                }
        }
}

Save the file. Next we will make some directories and get some more files.

sudo mkdir /var/www
cd /var/www
sudo wget https://raw.githubusercontent.com/arut/nginx-rtmp-module/master/stat.xsl
sudo chown www-data:www-data stat.xsl 

Next we will edit some nginx settings.

sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t
sudo service nginx reload
sudo mkdir sudo mkdir /var/www/wordpress -p

Next we will install PHP and MySQL.

sudo add-apt-repository ppa:ondrej/php5
sudo apt-get update
sudo apt-get install php5-common php5-mysqlnd php5-xmlrpc php5-curl php5-gd php5-cli php5-fpm php-pear php5-dev php5-imap php5-mcrypt
sudo su
apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A

echo "deb http://repo.percona.com/apt `lsb_release -cs` main" >> /etc/apt/sources.list.d/percona.list
echo "deb-src http://repo.percona.com/apt `lsb_release -cs` main" >> /etc/apt/sources.list.d/percona.list

apt-get update
exit
sudo apt-get install percona-server-server-5.6 percona-server-client-5.6

You might get 404 errors with the php PPA stuff.. It doesn’t officially support debian. Create a password for percona when it prompts.

mysql -u root -p -e 'create database wordpress'
mysql -u root -p -e "create user 'wordpress'@'localhost' identified by 'PASSWORD';"
mysql -u root -p -e "grant all privileges on wordpress.* to 'wordpress'@'localhost';"

Great, now we have a sql database configured for wordpress!

sudo nano /etc/nginx/sites-available/wordpress

Place the following information into the file. Change as necessary

server {
    listen 80;
    server_name _;
    access_log   /var/log/nginx/wordpress.access.log;
    error_log    /var/log/nginx/wordpress.error.log;
    root /var/www/wordpress;
    index index.php;
    location / {
            try_files $uri $uri/ /index.php?$args;
    }

    location ~ \.php$ {
            try_files $uri =404;
            include fastcgi_params;
            fastcgi_pass unix:/var/run/php5-fpm.sock;
    }

    location /viewers {
            rtmp_stat all;
            rtmp_stat_stylesheet stat.xsl;
    }
    location /stat.xsl {
            root /var/www/;
    }
    location /rec {
            autoindex on;
    }
}
sudo ln -s /etc/nginx/sites-available/wordpress /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-available/default
sudo nginx -t && sudo service nginx reload

Okay, we are pretty much finished with nginx configuration! Time for installing and customizing WordPress!

WordPress installation and configuration

I assume that you will be able to install WordPress on the server. There are probably hundreds of guides on the internet explaining it.

After installing it, you should be at your admin window. Follow THIS GUIDE I wrote earlier to customize WordPress for an RTMP frontend.

 

Additional Utilities and directories

After configuring WordPress, we still need to add some more utilities and directories to the server.

First we will create a directory to hold the scripts. These will be the heart of the server.

sudo mkdir /opt/nginx_scripts
sudo chown www-data:www-data -R /opt/nginx_scripts
cd /opt/nginx_scripts
sudo wget https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
sudo chmod +x wp-cli.phar 
sudo mv wp-cli.phar /usr/local/bin/wp
sudo apt-get install xmlstarlet ffmpeg
sudo mkdir /var/www/wordpress/assets
sudo mkdir /var/www/wordpress/rec
sudo mkdir /var/www/wordpress/rec/mp4
sudo mkdir /var/www/wordpress/jwplayer
cd /var/www/wordpress/jwplayer
cd /var/www/wordpress/jwplayer
sudo wget http://willstare.com/wp-content/uploads/2014/12/jwplayer.tar
sudo tar -xvf /var/www/wordpress/jwplayer/jwplayer.tar
sudo chown www-data:www-data -R /var/www/wordpress/assets
sudo chown www-data:www-data -R /var/www/wordpress/rec
sudo chown www-data:www-data -R /var/www/wordpress/jwplayer

Great, now we can add the scripts! First we will create the nginx_watcher script. This will listen for new RTMP streams, and then do some WordPress magic to create a user profile page with the stream information.

sudo nano /opt/nginx_scripts/nginx_watcher.sh

Paste this script in. Make sure the IP/URL’s are correct. Nothing will work unless they are correctly configured.

#!/bin/bash

#MASTER contains a list of registered usernames for the event. To clear it, simply delete the file!
MASTER="rtmp_users.list"

#Enter the IP address of the server. (ex. http://10.0.0.202)
IP_ADDRESS=10.0.0.10
#Enter the path to this script. Follow this syntax. Note the absence of a trailing '/' (ex: /home/user/scripts)
SCRIPT_PATH=/opt/nginx_scripts
#Enter the path to your WordPress installation. Follow this syntax. Note the absence of a trailing '/' (ex: /var/www/html)
WP_PATH=/var/www/wordpress
#Enter the URL of the Nginx statistics page. (ex: http://10.0.0.241/stats)
NGINX_STATS=http://10.0.0.10/viewers
#Enter the path to the HTML assets folder. This is where the server will hold the .m3u files. YOU MUST CHOWN this folder! (ex: /var/www/html/assets)
ASSETS=/var/www/wordpress/assets
#Enter the URL to the assets folder. (ex: http://10.0.0.241/assets)
WEB_ASSETS=http://10.0.0.10/assets
#Enter the path to the recording folder. (ex: /var/www/wordpress/rec/mp4)
REC=/var/www/wordpress/rec/mp4
#Enter the rec path. This is where the recordings will be stored. (ex: http://10.0.0.202/rec/mp4)
REC_PATH=http://10.0.0.10/rec/mp4
#Enter the RTMP server path. (ex: rtmp://10.0.0.242/live)
RTMP_PATH=rtmp://10.0.0.10/live
#Optional: Enter the path to a background image for JWPLayer. (ex: http://10.0.0.241/assets/site_logo.png)
IMAGE_PATH=
#Changes directory to the script-path for execution (required for cron)
cd $SCRIPT_PATH
#Creating the master list file.
MASTER="rtmp_users.list"
/bin/touch $MASTER
/bin/touch toadd.list
/bin/rm toadd.list
/bin/touch toadd.list
#Getting the current statistics from the streaming server
wget -O stats.xml $NGINX_STATS
#Parsing the XML file to get all the handles/keys currently streaming. These should correspond with the usernames of the streamers.
xmlstarlet sel -t -m '//name' -v . -n <stats.xml | grep -v live > unsanitized.list
sed -r 's/^\s*(.*\S)*\s*$/\1/;/^$/d' unsanitized.list > livestreams.list
#The loop below cycles through the livestreams.list file and compares each line to the current master list. If it doesn't exist, it adds it to the master list. It also adds it to the list of things we need to add to the website.
while read NAME
do
if ! /bin/grep -Fxq $NAME $MASTER; then
echo $NAME >> $MASTER
echo $NAME >> toadd.list
fi
done <livestreams.list
#Getting the existing post titles and ID's from WordPress. These will be compared to the currently active streams.
/usr/local/bin/wp --path=$WP_PATH post list --post_type=page --fields=ID,post_title > $SCRIPT_PATH/id_title.list
#Loop to find names to add. If they don't exist, a new page is created for them. This will display at the top of the menu. It is currently empty at this stage. Also checks and makes a blog post that will feature recordings from the user.

while read NEW; do
if ! /bin/grep $NEW id_title.list; then
/usr/local/bin/wp --path=$WP_PATH post create --post_type=page --post_status=publish --post_title=$NEW
/usr/local/bin/wp --path=$WP_PATH post create --post_type=post --post_status=publish --post_title=$NEW" Recordings"
fi
done <$MASTER

#The loop below checks to see if the stream is currently live or dead (depends on if its in the livestreams.list file). If its live, it generates a page with the relavent
#stream information customized for the user. If its dead, it will generate a dead page to notify browsers that the user is not currently streaming. This loop will update
# ALL streams in the master_rtmp.list as often as the cronjob is executed.
while read STREAM; do
echo $STREAM
/bin/ls $REC | grep $STREAM > $STREAM.record.list
cat $STREAM.record.list

POST_ID=`/usr/local/bin/wp --path=$WP_PATH post list --fields=ID,post_title | /bin/grep $STREAM | /usr/bin/awk '{print $1}'`
/bin/echo "<html>" >> $STREAM.record.html
/bin/echo "<head>" >> $STREAM.record.html
/bin/echo "<!--more-->" >> $STREAM.record.html
/bin/echo "</head>" >> $STREAM.record.html
/bin/echo "<body>" >> $STREAM.record.html
/bin/echo "<p>EXPERIMENTAL! Use Chrome or Firefox. Fullscreen for higest quality. If there is nothing here, "$STREAM" doesn't have any recordings. Page is regenerated every 3 minutes</p>" >> $STREAM.record.html
while read LINKS; do
/bin/echo "<a href="$REC_PATH/$LINKS">"$LINKS"</a>" >> $STREAM.record.html
/bin/echo "<video width='100%' controls>" >> $STREAM.record.html
/bin/echo "<source src='$REC_PATH/$LINKS' type='video/mp4'>" >> $STREAM.record.html
/bin/echo "Your browser does not support the HTML5 video player. Please download the file and play locally. Thank you." >> $STREAM.record.html
/bin/echo "</video>" >> $STREAM.record.html
done <$STREAM.record.list
/bin/echo "</body>" >> $STREAM.record.html
/bin/echo "</html>" >> $STREAM.record.html
RECORD_HTML=`cat $STREAM.record.html`
echo "ID BELOW"
echo $POST_ID

/usr/local/bin/wp --path=$WP_PATH post update $POST_ID --post_content="$RECORD_HTML"

ID=`/usr/local/bin/wp --path=$WP_PATH post list --post_type=page --fields=ID,post_title | /bin/grep $STREAM | /usr/bin/awk '{print $1}'`
/bin/rm $STREAM.record.list

if /bin/grep $STREAM livestreams.list; then
echo $STREAM" is online"
/bin/echo $RTMP_PATH/$STREAM > $ASSETS/$STREAM.m3u
/bin/echo "<html>" >> $STREAM.livetemplate.html
/bin/echo "<head>" >> $STREAM.livetemplate.html
/bin/echo "<script src=http://"$IP_ADDRESS"/jwplayer/jwplayer.js></script>" >> $STREAM.livetemplate.html
/bin/echo "</head>" >> $STREAM.livetemplate.html
/bin/echo "<body>" >> $STREAM.livetemplate.html
/bin/echo "<div id="$STREAM"></div>" >> $STREAM.livetemplate.html
/bin/echo "<script>// <![CDATA[" >> $STREAM.livetemplate.html
/bin/echo "jwplayer('"$STREAM"').setup({ file: '"$RTMP_PATH/$STREAM"', image: '//"$IMAGE_PATH"', title: '"$STREAM"', width: '100%', aspectratio: '16:9' });" >> $STREAM.livetemplate.html
/bin/echo "// ]]></script>" >> $STREAM.livetemplate.html
/bin/echo $STREAM" is online and streaming as of "`date` >> $STREAM.livetemplate.html
/bin/echo "<a href="$WEB_ASSETS/$STREAM".m3u>VLC LINK</a> (Right Click and Save-as)" >> $STREAM.livetemplate.html
/bin/echo "<p></p>" >> $STREAM.livetemplate.html
/bin/echo "<h1><a href="http://$IP_ADDRESS"/"$STREAM"-recordings>"$STREAM"'s past recordings</a></h1>" >> $STREAM.livetemplate.html
/bin/echo "</body>" >> $STREAM.livetemplate.html
/bin/echo "</html>" >> $STREAM.livetemplate.html
LIVE_HTML=`cat $STREAM.livetemplate.html`
/usr/local/bin/wp --path=$WP_PATH post update $ID --post_content="$LIVE_HTML"
/bin/rm $STREAM.livetemplate.html

else
/bin/echo $STREAM" is offline"
/bin/echo "<html>" >> $STREAM.deadtemplate.html
#/bin/echo "<head></head>" >> $STREAM.deadtemplate.html
/bin/echo "<h1>"$STREAM" is not streaming as of "`date`" </h1>" >> $STREAM.deadtemplate.html
/bin/echo "<body>" >> $STREAM.deadtemplate.html
#/bin/echo "<p></p>" >> $STREAM.deadtemplate.html
/bin/echo "<h1><a href="http://$IP_ADDRESS"/"$STREAM"-recordings>"$STREAM"'s past recordings</a></h1>" >> $STREAM.deadtemplate.html
/bin/echo "Page is regenerated every 60 seconds, please check again shortly." >> $STREAM.deadtemplate.html
/bin/echo "If this is your stream and it should be running, please contact an event administrator." >> $STREAM.deadtemplate.html #This string can be modified to meet your requirements.
/bin/echo "</body>" >> $STREAM.deadtemplate.html
/bin/echo "</html>" >> $STREAM.deadtemplate.html
DEAD_HTML=`cat $STREAM.deadtemplate.html`
/usr/local/bin/wp --path=$WP_PATH post update $ID --post_content="$DEAD_HTML"
/bin/rm $STREAM.deadtemplate.html
fi
/bin/rm $STREAM.record.html
done <$MASTER
#CLEANUP to remove some temp files that we don't need. These could be converted into arrays at a later point, but it makes it very simple to use files/grep.
/bin/rm stats.xml
/bin/rm id_title.list
/bin/rm livestreams.list
/bin/rm unsanitized.list
/bin/rm toadd.list
/bin/rm *.record.html
#This last line is useful for logging. If you run this script as a cronjob and send the output to a log file, you will see when the last time the script ran.
/bin/echo "Script finished at" `date`

Then we will create the recording_watcher.sh script. This will check for new recordings, and if it finds any, it will process them and move them to the web server directory. From there, users can view them.

sudo nano /opt/nginx_scripts/recording_watcher.sh
#!/bin/bash
#This script watches the /opt/ngnix_rtmp/live_recording directory for .flv files and converts them to .mp4 to make sharing the videos easier within a web browser.

#Enter the path to the script (ex: /opt/nginx_scripts)
SCRIPT_PATH=/opt/nginx_scripts
#Enter the path that NGINX-RTMP is saving files to. (ex: /opt/nginx_scripts/live_recording)
RTMP_RECORD=/opt/nginx_scripts/live_recording
#Enter the path to the webserver directory that the videos will be placed into. (ex: /var/www/wordpress/rec/mp4)
REC=/var/www/wordpress/rec/mp4

cd $RTMP_RECORD
#List all files in the /opt/nginx_rtmp/live_recording directory that match .flv. It writes the list to the flash_vids.list file with the full path.
ls -d -1 $PWD/** | grep .flv > $SCRIPT_PATH/flash_vids.list
#lsof is used to find what files are currently open by the OS. We use it here to see what files are open in the /opt/nginx_rtmp/live_recordings directory that match .flv. If it finds a
#file currently open, it places the name of the file into the writing_vids.list file.
lsof $RTMP_RECORD/* | grep .flv | /usr/bin/awk '{print $9}' > $SCRIPT_PATH/writing_vids.list
cd $SCRIPT_PATH

#Loop below reads through the writing_vids.list file and cross references it to the flash_vids.list file. If there's a match, it removes the active file from the flash_vids.list file.
#These files will not be processed in the next loop.
while read WRITING; do
grep -v $WRITING flash_vids.list > flash.tmp
mv flash.tmp flash_vids.list
done <writing_vids.list

#Sed strips the full path from the file names within the flash_vids.list file.
#!!!!!!!!!!!!!!!!!! ENTER THE CORRECT PATH BELOW. SHOULD BE THE SAME AS RTMP_RECORD VARIABLE. THANK YOU !!!!!!!!!!!!!!!!!!!!!!!!!!!

sed -i 's/\/opt\/nginx_scripts\/live_recording\///g' flash_vids.list
#Loop below reads through the flash_vids.list file and processes each .flv with avconv. It losslessly (and quickly!) converts the .flv video file to .mp4. It then removes the .flv pattern
#from the filenames. At the end it removes the .flv file.
while read FLASH; do
/usr/bin/avconv -i $RTMP_RECORD/$FLASH -vcodec copy -acodec copy $RTMP_RECORD/$FLASH.mp4
/usr/bin/rename 's/.flv//g' $RTMP_RECORD/$FLASH.mp4
/bin/rm $RTMP_RECORD/$FLASH
done <flash_vids.list

#Cleanup. Move all .mp4 files to the /var/www/wordpress/rec/mp4/ directory, and deletes the flash_vids.list and writing_vids.list files.
/bin/mv $RTMP_RECORD/*.mp4 $REC
/bin/rm flash_vids.list
/bin/rm writing_vids.list

Save that file as well. Now you should have 2 scripts, nginx_watcher and recording_watcher. Now we need to assign some final permissions and ownerships, and create a cron job to run every 60 seconds.

sudo chmod a+x nginx_watcher.sh
sudo chmod a+x recording_watcher.sh
sudo chown www-data:www-data nginx_watcher.sh
sudo chown www-data:www-data recording_watcher.sh
sudo mkdir /opt/nginx_scripts/live_recording
sudo chown www-data:www-data /opt/nginx_scripts/live_recording

sudo -u www-data -s #change to the www-data user to add a cronjob. 
./nginx_watcher.sh
./recording_watcher.sh
#those should both run.
crontab -e
*/1 * * * * /opt/nginx_scripts/nginx_watcher.sh >> /opt/nginx_scripts/nginx_watcher.log
*/3 * * * * /opt/nginx_scripts/recording_watcher.sh >> /opt/nginx_scripts/recording_watcher.log
touch /opt/nginx_scripts/nginx_watcher.log
touch /opt/nginx_scripts/recording_watcher.log
exit

If all went well, you should now have a fully functional Nginx-RTMP + WordPress installation capable of handling streaming sources and clients. It should be able to record all streams running through it.

Troubleshooting Tips

  • Make sure paths are correct in all scripts, symlinks, etc
  • Doublecheck permissions, especially in the www folder and the scripts folder
  • Make sure all paths exist
  • If you’re having problems, check the log for nginx and the script
  • Run the scripts and check for errors in their output
  • If all else fails, leave a comment below.

If you have a large event coming up and would like to use this for your streaming needs, please feel free to email me at will [@] willstare.com. I can work with you to provide assistance, or even meet additional requirements.

There may be errors in this guide. If you spot any, please leave a comment. Thanks!

Local Nginx RTMP server v2.0

After taking a break from the project, I am back with version 2!

The biggest change is that it is now based on a single Debian VM instead of 2 Ubuntu machines. Switching to a single VM will use less system resources and make iy much easier to implement on a single physical box.

There is also a new feature: Recording! Now every stream is archived for later viewing, allowing tournaments and other events to be saved for later distribution. These recordings are incorporated into the WordPress site, and allow users to easily browse saved streams.

I will begin a new series going over the installation and configuration of the Debian VM, and again will provide a download link for the finished product. If you have already implemented the dual-machine solution, I’d recommend you take a look at the single-machine setup.

RTMP WordPress VM download

VIRTUAL MACHINE DOWNLOAD

I made 2 virtual machines (VMware workstation 8.0 compatable) that are ready for deployment in your LAN. You’ll want to run these on a decent ESXi or VMware workstation host with a 1gbps or higher connection. Just download these machines and import them. You’ll probably want to assign a static IP address to these before you send them into production. On the WordPress website you’ll want to change a few of the links to match up with your system’s IP addresses.

Both VMs are running up to date (as of 2014-09-21) Ubuntu 14.04 64bit. The streaming server is set for 4 cores, so if your VM host only has a dual core you should adjust that. They require a combined 3gb of ram.

They are fully configured, and WordPress is installed and customized exactly how the guide shows.

Credentials for both (you should probably change these)

Username: streamer
Password: dh2n7

These will work for logging into the operating system, mysql/phpmyadmin root account, and WordPress admin account (wpress database for the website will use a different password).

WordPress credentials:

Database username: wpress
Database password: m3c6x

Again, you should probably change these!

VIRTUAL MACHINE DOWNLOAD

Finishing the RTMP WordPress project

If you have been following these tutorials, you should now have a fully functional Nginx RTMP server as well as a WordPress site optimized for showing live streams.

We just need to install a few more things and this project will be complete!

The first package we need is wp-cli. This program will allow us to modify our WordPress site from the CLI! Cool!

 user@websvr:~$ wget https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar 

Check to see if it works by typing

 user@websvr:~$ php wp-cli.phar --info 

If it works, we can make it an executable so you just type “wp” to call it.

user@websvr:~$ chmod +x wp-cli.phar

user@websvr:~$ sudo mv wp-cli.phar /usr/local/bin/wp 

That’s all we have to do to enable wp-cli. Next we need to install xmlstarlet, a utility to parse xml files.

 user@websvr:~$ sudo apt-get install xmlstarlet 

After that installs, we are finished adding software to the server.  Everything else should be included in your distro.

The script we will be using automatically generates .m3u playlist files that will need a directory in the webserver to reside. I like to use an ‘assets’ folder, but you can name it anything you’d like. Make a folder in the www root of your webserver, where WordPress is installed. After we create the folder, we must ‘chown’ it, which means change owner. Change ownership to the user that will be running the script.

 user@websvr:~$ sudo mkdir /var/www/html/assets/

user@websvr:~$ sudo chown user /var/www/html/assets/ 

 

What follows is a custom script that will allow the WordPress site to update based on the status of the nginx RTMP server. It will require some input before you run it, but once its set it can take care of itself.


#!/bin/bash

#MASTER contains a list of registered usernames for the event. To clear it, simply delete the file!
MASTER="rtmp_users.list"

#Enter the path to this script. Follow this syntax. Note the absence of a trailing '/' (ex: /home/user/scripts)
SCRIPT_PATH=
#Enter the path to your WordPress installation. Follow this syntax. Note the absence of a trailing '/' (ex: /var/www/html)
WP_PATH=
#Enter the URL of the Nginx statistics page. (ex: http://10.0.0.241/stats)
NGINX_STATS=
#Enter the path to the HTML assets folder. This is where the server will hold the .m3u files. YOU MUST CHOWN this folder! (ex: /var/www/html/assets)
ASSETS=
#Enter the URL to the assets folder. (ex: http://10.0.0.241/assets)
WEB_ASSETS=
#Enter the RTMP server path. (ex: rtmp://10.0.0.242/live)
RTMP_PATH=
#Optional: Enter the path to a background image for JWPLayer. (ex: http://10.0.0.241/assets/site_logo.png)
IMAGE_PATH=
#Changes directory to the script-path for execution (required for cron)
cd $SCRIPT_PATH
#Creating the master list file.
MASTER="rtmp_users.list"
/bin/touch $MASTER
/bin/touch toadd.list
/bin/rm toadd.list
/bin/touch toadd.list
#Getting the current statistics from the streaming server
wget -O stats.xml $NGINX_STATS
#Parsing the XML file to get all the handles/keys currently streaming. These should correspond with the usernames of the streamers.
xmlstarlet sel -t -m '//name' -v . -n <stats.xml | grep -v live > livestreams.list
#The loop below cycles through the livestreams.list file and compares each line to the current master list. If it doesn't exist, it adds it to the master list. It also adds it to the list of things we need to add to the website.
while read NAME
do
if ! /bin/grep -Fxq $NAME $MASTER; then
echo $NAME >> $MASTER
echo $NAME >> toadd.list
fi
done <livestreams.list
#Getting the existing post titles and ID's from WordPress. These will be compared to the currently active streams.
/usr/local/bin/wp --path=$WP_PATH post list --post_type=page --fields=ID,post_title > $SCRIPT_PATH/id_title.list
#Loop to find names to add. If they don't exist, a new page is created for them. This will display at the top of the menu. It is currently empty at this stage.
while read NEW; do
if ! /bin/grep $NEW id_title.list; then
/usr/local/bin/wp --path=$WP_PATH post create --post_type=page --post_status=publish --post_title=$NEW
fi
done <$MASTER
#The loop below checks to see if the stream is currently live or dead (depends on if its in the livestreams.list file). If its live, it generates a page with the relavent
#stream information customized for the user. If its dead, it will generate a dead page to notify browsers that the user is not currently streaming. This loop will update
# ALL streams in the master_rtmp.list as often as the cronjob is executed.
while read STREAM; do
ID=`/usr/local/bin/wp --path=$WP_PATH post list --post_type=page --fields=ID,post_title | /bin/grep $STREAM | /usr/bin/awk '{print $1}'`
if /bin/grep $STREAM livestreams.list; then
echo $STREAM" is online"
/bin/echo $RTMP_PATH/$STREAM > $ASSETS/$STREAM.m3u
/bin/echo "<html>" >> $STREAM.livetemplate.html
/bin/echo "<head>" >> $STREAM.livetemplate.html
/bin/echo "<script src=http://jwpsrv.com/library/UjjVOi7iEeSiiiIAC0MJiQ.js></script>" >> $STREAM.livetemplate.html
/bin/echo "</head>" >> $STREAM.livetemplate.html
/bin/echo "<body>" >> $STREAM.livetemplate.html
/bin/echo "<div id="$STREAM"></div>" >> $STREAM.livetemplate.html
/bin/echo "<script>// <![CDATA[" >> $STREAM.livetemplate.html
/bin/echo "jwplayer('"$STREAM"').setup({ file: '"$RTMP_PATH/$STREAM"', image: '//"$IMAGE_PATH"', title: '$NEW', width: '100%', aspectratio: '16:9' });" >> $STREAM.livetemplate.html
/bin/echo "// ]]></script>" >> $STREAM.livetemplate.html
/bin/echo $STREAM" is online and streaming as of "`date` >> $STREAM.livetemplate.html
/bin/echo "<a href="$WEB_ASSETS/$STREAM".m3u>VLC LINK</a> (Right Click and Save-as)" >> $STREAM.livetemplate.html
/bin/echo "</body>" >> $STREAM.livetemplate.html
/bin/echo "</html>" >> $STREAM.livetemplate.html
LIVE_HTML=`cat $STREAM.livetemplate.html`
/usr/local/bin/wp --path=$WP_PATH post update $ID --post_content="$LIVE_HTML"
/bin/rm $STREAM.livetemplate.html
else
/bin/echo $STREAM" is offline"
/bin/echo "<html>" >> $STREAM.deadtemplate.html
/bin/echo "<head></head>" >> $STREAM.deadtemplate.html
/bin/echo "<h1>"$STREAM" is not streaming as of "`date`" </h1>" >> $STREAM.deadtemplate.html
/bin/echo "<body>" >> $STREAM.deadtemplate.html
/bin/echo "Page is regenerated every 60 seconds, please check again shortly." >> $STREAM.deadtemplate.html
/bin/echo "If this is your stream and it should be running, please contact an event administrator." >> $STREAM.deadtemplate.html #This string can be modified to meet your requirements.
/bin/echo "</body>" >> $STREAM.deadtemplate.html
/bin/echo "</html>" >> $STREAM.deadtemplate.html
DEAD_HTML=`cat $STREAM.deadtemplate.html`
/usr/local/bin/wp --path=$WP_PATH post update $ID --post_content="$DEAD_HTML"
/bin/rm $STREAM.deadtemplate.html
fi
done <$MASTER
#CLEANUP to remove some temp files that we don't need. These could be converted into arrays at a later point, but it makes it very simple to use files/grep.
/bin/rm stats.xml
/bin/rm id_title.list
/bin/rm livestreams.list
/bin/rm toadd.list
#This last line is useful for logging. If you run this script as a cronjob and send the output to a log file, you will see when the last time the script ran.
/bin/echo "Script finished at" `date` 

Create a folder in the home directory of the user that will be running the script. Inside that folder, create a new file.

 user@websvr:~$ nano nginx_watcher.sh 

Edit the file

Paste the script into this file, and save it. Make the file executable by typing

 user@websvr:~$ chmod +x nginx_watcher.sh 

 

We are almost finished! Start a stream to the Nginx-RTMP server using your streaming software.  Then on the webserver, run the script by typing

 user@websvr:~$ ./nginx_watcher.sh 

If all goes well, there will be no errors reported, and your WordPress site will have a new page set up for the stream. The live video should be playable through JWPlayer, and an m3u file will be available for download. You should also be able to stop the stream, run the script again, and the page will be updated to state the the stream is offline.

The last step is to create a cronjob for this script to run every minute. This will allow users to begin streaming and a page for them will be created no more than 1 minute after.

Type

 user@websvr:~$ crontab -e 

and create a job run every minute. Here’s mine:

 */1 * * * * /home/user/nginx_water/nginx_watcher.sh >> /home/user/nginx_watcher/log/nginx_watcher.log 2>&1 

That will run the script every minute and send the output to a log file.

After that, your Nginx-RTMP/WordPress Frankenstein should be ready to roll!

Screenshots of it working!

Screenshot of the webpage when someone is actively streaming.

Screenshot of the webpage when someone is actively streaming.

Screenshot of the webpage when that same person has stopped streaming.

Screenshot of the webpage when that same person has stopped streaming.

I plan on releasing some prebuilt virtual machines soon for download for those who just want to get something running. THEY’RE HERE!

If you have any questsions, please leave a comment  below. If you have any suggestions for the script, I’d love to hear them. Thanks for reading.

Customizing WordPress for RTMP

m4s0n501

If you are following along in this mini-series, you should now have a fully functional nginx based RTMP server capable of receiving and broadcasting live video streams. As cool as that is, its not very user friendly, especially for potential viewers.

In this post we will go over the steps to get a frontend set up for our RTMP server. This will serve as a central hub for our streaming environment. It will host information about how to watch the streams, how to broadcast to the server, as well as some cool statistics. You will need the following.

  • Fully functional WordPress installation – This guide will not cover WordPress installation or configuration. There are dozens of quality guides on the internet for this. Basically you need an Ubuntu server running the LAMP stack.

So, let’s start!

Log into your fully up to date WordPress installation via the web browser. I have found a few customization options that make the website look better for streaming.

  1. Change the theme to “TwentyTwelve”. Its a simple, clean theme that won’t distract viewers.2012
  2. Remove all widgets from the side panel. This will clean up the pages even more. Removing the “meta” widget will remove the “login” link on the side of pages. To login in the future, you will have to go to your WordPress admin URL (ex. 10.0.0.242/wp-admin) .widgets
  3. Create a menu via the Appearance menu. In the menu options, click the checkbox for “Automatically add new top-level pages to this menu”. Also select Primary Menu. This will automatically make our links on the website when the script makes new pages. menu
  4. Create some informational pages for your viewers/streamers to learn about how to use the website to stream/watch streams. Here’s some screenshots of mine.
    How to stream

    A basic how-to guide on streaming to the RTMP server with OBS. This page also includes directions for Xsplit as well.

    statsfor nerds

    A simple iframe within a page that shows the RTMP stats. Not necessary, but its kind of cool :)

how to watch

A basic how to watch page with download links for VLC.

Install nginx-rtmp server

For this guide I am following this post.

Requirements

For this guide you will need an up to date Ubuntu Server 14.04 virtual or physical machine.

Update your operating system. If you’re on Ubuntu, its a simple process. Simply type

 user@websvr:~$ sudo apt-get update 

After that, just type

 user@websvr:~$ sudo apt-get upgrade 

This will download updates for all the installed software. When that finishes, we need to grab the dependencies for nginx. Type or paste the following into the server.

 user@streamsvr:~$ sudo apt-get install build-essential libpcre3 libpcre3-dev libssl-dev

Now that we have the dependencies, we can download the source code of nginx from their webpage.  As of this post, the most recent version is 1.7.4. To download it, type the following.

 user@streamsvr:~$ wget http://nginx.org/download/nginx-1.7.4.tar.gz

We also need to grab the RTMP module because it isn’t included in the main nginx distribution.

 user@streamsvr:~$ wget https://github.com/arut/nginx-rtmp-module/archive/master.zip 

Before we continue, we need to install the unzip program.

 user@streamsvr:~$ sudo apt-get install unzip 

Now we open the archives we downloaded and get cracking!

 user@streamsvr:~$ tar -zxvf nginx-1.7.4.tar.gz
user@streamsvr:~$ unzip master.zip
user@streamsvr:~$ cd nginx-1.7.4/ 

We can now configure the source and build the binary.

user@streamsvr:~/nginx-1.7.4$ ./configure --with-http_ssl_module --add-module=../nginx-rtmp-module-master
user@streamsvr:~/nginx-1.7.4$ make
user@streamsvr:~/nginx-1.7.4$ sudo make install

Now nginx is installed! It is install ed in /usr/local/nginx. To start it type

 user@streamsvr:~$ sudo /usr/local/nginx/sbin/nginx 

Test it by going to your server IP with your web browser. It should display the following:

nginx installed

We are almost done. To add RTMP functionality, we must enable it in the nginx configuration file.

 user@streamsvr:~$ sudo nano /usr/local/nginx/conf/nginx.conf 

Paste this in at the end of the file.

rtmp {
        server {
                listen 1935;
                chunk_size 4096;

                application live {
                        live on;
                        record off;
                }
        }
}

To save, hit Crtl+x , yes, and enter.

To make the rest of the project easier, we should make nginx into a system service that runs automatically and can be controlled with the service command.

 sudo nano /etc/init/nginx.conf 

Paste this into the file

# nginx

description "nginx http daemon"
author "Philipp Klose <me@'thisdomain'.de>"

start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]

env DAEMON=/usr/local/nginx/sbin/nginx
env PID=/var/run/nginx.pid

expect fork
respawn
respawn limit 10 5
#oom never

pre-start script
 $DAEMON -t
 if [ $? -ne 0 ]
 then exit $?
 fi
end script

exec $DAEMON

We can now start and stop nginx with the sudo service nginx start | stop | restart commands. Yes! It will also start when the server boots, so you don’t need to worry about that.

Next we need to go back to the nginx configuration file and prepare it for stats delivery. The ability to display server statistics will be crucial to this project. This page will ensure that live streams have the correct pages generated for them.

Go back to the nginx configuration and add the following inside the server directive in the /usr/local/nginx/conf/nginx.conf.

# rtmp statistics
        location /viewers {
            rtmp_stat all;
            rtmp_stat_stylesheet stat.xsl;
        }

        location /stat.xsl {
            # you can move stat.xsl to a different location
            # under linux you could use /var/user/www for example
            root html;
}

The server directive should now start like this:


server {
listen 80;
server_name localhost;

#charset koi8-r;

#access_log logs/host.access.log main;

location / {
root html;
index index.html index.htm;
}
#rtmp statistics
location /viewers {
rtmp_stat all;
rtmp_stat_stylesheet stat.xsl;
}

location /stat.xsl {
# you can move stat.xsl to a different location
# under linux you could use /var/user/www for example
root html;
}
#error_page 404 /404.html; 

We have to do one more quick thing before the stats page will work. In order to generate the XML, the RTMP module uses an XSL file. This will need to be placed in the HTML root of the nginx server. Create a file called stat.xsl containing the following. Save it, and we should be good to go. For whatever reason, pasting the stat.xsl file into this page is messing things up, so I’ll just include a link to the file. 

 user@streamsvr:~$ sudo nano /usr/local/nginx/html/stat.xsl 

After you paste that in, save the nginx configuration. Restart the entire server. After it boots should be able to see the rtmp server stats if you go to your server IP address /viewers.

And that’s it for the nginx-rtmp installation/configuration. The server is now running and ready for the next section of this series… Web server configuration and scripting!

 

 

 

 

 

Local RTMP server with Website

I am starting a mini-series of posts on how to set up Nginx-RTMP and WordPress to create a local version of Twitch.tv. I will post detailed guides on installing and configuring all software required for this project.

Project requirements are as follows:

  • 1 or 2 Linux computers/VMs. All testing has been done with Ubuntu Server 14.04 64bit.
  • Nginx-RTMP with the stats plugin compiled.
  • A working WordPress installation. (not covered in this series)
  • Various command line utilities that will be covered in depth during the guide.

The end product will be a fully functional streaming server capable of handling 1gbps of video traffic, with a custom WordPress frontend for users to use to access the streams. The entire system is automated and requires minimal user input. A basic use-case is below.

Bob wants to stream his Dota game for the entire LAN party to see. The LAN has a limited bandwidth internet connection, so using a service like Twitch.tv is out of the question. However, the LAN execs have configured a local RTMP solution for him to use.

Bob configures his streaming software like he would normally, except he points his stream to the local RTMP server. He uses his gamer handle (b0b) as a stream key. At this point, Bob’s job is finished. He can start playing his game.

On the backend, a script reads that a new stream has started. It checks to see if this stream has been played before, or if this is a new stream all together. If its new, it adds the stream to the local database and creates a page on the WordPress site dedicated to Bob’s stream.  It then runs through every stream that has been registered to the server during the event. If the stream is active, like Bob’s, it updates the page with a JWPlayer configured specifically for Bob’s stream. It also generates an .m3u file with a playlist entry pointing towards Bob’s stream mount point.  Viewers can just navigate to Bob’s page on the WordPress site and start watching from within their web browser.

If some of the previously registered streams are offline, it updates their page with an offline message telling users to check back soon. By using this process, every single page is updated every 60 seconds via a cronjob.

So that’s a basic overview of what this setup can do.  It can listen for, create, and distribute live RTMP streams to viewers automatically and painlessly. And because all traffic is local, no internet bandwidth is wasted for redundant connections to websites like Twitch.tv or hitbox.

I will be making many more posts about how to set this up for your own event, so stay tuned. If you have any questions or comments, please leave a message below.

Part 1 – Install nginx rtmp tutorial here!

Part 2 – Customize WordPress tutorial here!

Part 3 – Finishing the RTMP WordPress Project here!

Part 4 – Virtual Machine Download here!