I have created a demo virtual machine that will allow users to try out the system. It is fully functional, but will require a good deal of configuration change if you want to implement it in an environment using IP addresses not in the 10.x.x.x network. There is a readme that I highly recommend you read before using it.
After working with Nginx-RTMP, I wanted to see how much traffic I could actually put through my server setup. I tried just streaming a bunch of video feeds from the server, but I ran out of processing power on my desktop. I had to figure out a better solution.
Enter Flazr. Flazr is a Java application that allows you to, among other things, stress test an RTMP server.
What you’ll need is:
For basic usage, just use the command line to initiate the test. Here’s an example of a configuration I used.
./client.sh -load 100 rtmp://10.0.0.10/live/willstare
The stream I was pulling was at 1500kbps. Here’s the bandwidth being used on the RTMP server.
rx: 6.20 Mbit/s 8944 p/s tx: 169.46 Mbit/s 18667 p/s
And here’s the CPU usage on the RTMP server.
load average: 0.01, 0.02, 0.05
Obviously, my server can handle more than 100 connections. Let’s crank it up to 675! 675*1.5mbps = 975mbps.. That’s just about hitting the top end of gigabit.
Before we do that, we should probably increase the heap size for Flazr. On Ubuntu, type nano cleint.sh to edit the script. Change -Xmx512m to -Xmx2048m (if you have at least 4gb of RAM in the machine).
./client.sh -load 675 rtmp://10.0.0.10/live/willstare
Server bandwidth @ 675 clients
rx: 6.04 Mbit/s 8655 p/s tx: 961.13 Mbit/s 81735 p/s
And server load @ 675 clients
load average: 0.07, 0.03, 0.05
As you can see, RTMP streaming is very efficient. It only requires massive bandwidth. Even with 675 clients, the video was still smooth when viewing it from another machine.
If you’re like me, you’ve got a shoutcast server configured to handle thousands of connections. I was pretty sure my setup could handle the load, but I wanted to actually find out. After looking around online, I found out an easy way to stress test the server using curl. Here’s a little script I wrote that will allow you to murder your server :D. Run this script on a test machine with a decent internet connection so you can really stress the server. This script should run on basically any linux operating system. I used Ubuntu as my test box.
Just a few things before you run the script.. You will need vnstat (or you’ll have to comment it out) to get the most of the script. Also, be careful with how many threads you run. If your server has the ulimit too low, you could lock up the shoutcast server.
#!/bin/bash
echo "enter a URL to stress test. ex: http://radio.derp.com:8123"
read URL
echo "enter number of threads to run. ex: 50"
read THREADS
for ((N=0; N<$THREADS; N++))
do curl -o /dev/null $URL >> /dev/null 2>&1 &
done
echo "Created "$THREADS" threads connected to "$URL
echo "Live Bandwidth"
echo "hit ctrl-c when you are finished"
vnstat -l
echo "Press any button and hit enter to kill all curl processes! Thanks!"
read Q
killall -e curl
If you have any questions please leave a comment below. Thanks
Here’s an example of the script in action.
./shoutcast_stress.sh enter a URL to stress test. ex: http://radio.derp.com:8123 stream.willstare.com:8123 enter number of threads to run. ex: 50 1500 Created 1500 threads connected to stream.willstare.com:8123 Live Bandwidth hit ctrl-c when you are finished Monitoring em0... (press CTRL-C to stop) rx: 50.45 Mbit/s 4514 p/s tx: 2.48 Mbit/s 4503 p/s
And bandwidth usage on the server
rx: 4.14 Mbit/s 7706 p/s tx: 66.88 Mbit/s 8105 p/s
And what Shoutcast looks like with 1500+ connections:
There’s a few glitches (backspace might now work when entering stream URL), but I think its a good tool.
Alright, let’s get started. This guide assumes you meet the following prerequisites:
Everything can be done to a physical installation or a virtual machine.
First we will install nginx-rtmp from a PPA. These steps are taken from HERE.
sudo apt-get install dpkg-dev
sudo mkdir /usr/src/nginx
cd /usr/src/nginx
sudo apt-get source nginx
sudo apt-get install git-core
sudo git clone https://github.com/arut/nginx-rtmp-module.git
cd nginx-1.2.1/
– Note: This will depend on what version you have. As of this writing, nginx was at 1.2.1.
List the directory. The following files should be shown in /usr/src/nginx/nginx-1.2.1/
sudo nano debian/rules
Add the following to the add-modules configuration.
module=/usr/src/nginx/nginx-rtmp-module \
So it looks like this
--add-module=$(MODULESDIR)/nginx-upstream-fair \
--add-module=$(MODULESDIR)/nginx-dav-ext-module \
--add-module=/usr/src/nginx/nginx-rtmp-module \
sudo apt-get build-dep nginx
sudo dpkg-buildpackage -b
Building the package may take a while.
cd ..
sudo dpkg -i nginx-common_1.2.1-2.2+wheezy3_all.deb nginx-full_1.2.1-2.2+wheezy3_amd64.deb
sudo service nginx start
Great! Now we have nginx-rtmp installed on our Debian installation. Next, we will configure some stuff within Nginx!
Start by changing nginx worker processes to 1. This is to make sure the streaming stats don’t get all screwy.
sudo nano /etc/nginx/nginx.confSo it looks like this.
user www-data;
worker_processes 1;
pid /var/run/nginx.pid;
Then scroll down and add the following to the end of the file.
rtmp {
server {
listen 1935;
chunk_size 4096;
application live {
live on;
record all;
record_path /opt/nginx_scripts/live_recording/;
record_suffix -%d-%b-%y-%T.flv;
record_lock on;
}
}
}Save the file. Next we will make some directories and get some more files.
sudo mkdir /var/www
cd /var/www
sudo wget https://raw.githubusercontent.com/arut/nginx-rtmp-module/master/stat.xsl
sudo chown www-data:www-data stat.xsl Next we will edit some nginx settings.
sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t
sudo service nginx reload
sudo mkdir sudo mkdir /var/www/wordpress -pNext we will install PHP and MySQL.
sudo apt-get install php5-common php5-mysqlnd php5-xmlrpc php5-curl php5-gd php5-cli php5-fpm php-pear php5-dev php5-imap php5-mcrypt
sudo su
apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
echo "deb http://repo.percona.com/apt `lsb_release -cs` main" >> /etc/apt/sources.list.d/percona.list
echo "deb-src http://repo.percona.com/apt `lsb_release -cs` main" >> /etc/apt/sources.list.d/percona.list
apt-get update
exit
sudo apt-get install percona-server-server-5.6 percona-server-client-5.6You might get 404 errors with the php PPA stuff.. It doesn’t officially support debian. Create a password for percona when it prompts.
mysql -u root -p -e 'create database wordpress'
mysql -u root -p -e "create user 'wordpress'@'localhost' identified by 'PASSWORD';"
mysql -u root -p -e "grant all privileges on wordpress.* to 'wordpress'@'localhost';"Great, now we have a sql database configured for wordpress!
sudo nano /etc/nginx/sites-available/wordpress
Place the following information into the file. Change as necessary
server {
listen 80;
server_name _;
access_log /var/log/nginx/wordpress.access.log;
error_log /var/log/nginx/wordpress.error.log;
root /var/www/wordpress;
index index.php;
location / {
try_files $uri $uri/ /index.php?$args;
}
location ~ \.php$ {
try_files $uri =404;
include fastcgi_params;
fastcgi_pass unix:/var/run/php5-fpm.sock;
}
location /viewers {
rtmp_stat all;
rtmp_stat_stylesheet stat.xsl;
}
location /stat.xsl {
root /var/www/;
}
location /rec {
autoindex on;
}
}sudo ln -s /etc/nginx/sites-available/wordpress /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-available/default
sudo nginx -t && sudo service nginx reloadOkay, we are pretty much finished with nginx configuration! Time for installing and customizing WordPress!
I assume that you will be able to install WordPress on the server. There are probably hundreds of guides on the internet explaining it.
After installing it, you should be at your admin window. Follow THIS GUIDE I wrote earlier to customize WordPress for an RTMP frontend.
After configuring WordPress, we still need to add some more utilities and directories to the server.
First we will create a directory to hold the scripts. These will be the heart of the server.
sudo mkdir /opt/nginx_scripts sudo chown www-data:www-data -R /opt/nginx_scripts cd /opt/nginx_scripts sudo wget https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar sudo chmod +x wp-cli.phar sudo mv wp-cli.phar /usr/local/bin/wp sudo apt-get install xmlstarlet ffmpeg sudo mkdir /var/www/wordpress/assets sudo mkdir /var/www/wordpress/rec sudo mkdir /var/www/wordpress/rec/mp4 sudo mkdir /var/www/wordpress/jwplayer cd /var/www/wordpress/jwplayer cd /var/www/wordpress/jwplayer sudo wget https://www.willstare.com/wp-content/uploads/2014/12/jwplayer.tar sudo tar -xvf /var/www/wordpress/jwplayer/jwplayer.tar sudo chown www-data:www-data -R /var/www/wordpress/assets sudo chown www-data:www-data -R /var/www/wordpress/rec sudo chown www-data:www-data -R /var/www/wordpress/jwplayer
Great, now we can add the scripts! First we will create the nginx_watcher script. This will listen for new RTMP streams, and then do some WordPress magic to create a user profile page with the stream information.
sudo nano /opt/nginx_scripts/nginx_watcher.sh
Paste this script in. Make sure the IP/URL’s are correct. Nothing will work unless they are correctly configured.
#!/bin/bash
#MASTER contains a list of registered usernames for the event. To clear it, simply delete the file!
MASTER="rtmp_users.list"
#Enter the IP address of the server. (ex. 10.0.0.10)
IP_ADDRESS=10.0.0.10
#Enter the path to this script. Follow this syntax. Note the absence of a trailing '/' (ex: /home/user/scripts)
SCRIPT_PATH=/opt/nginx_scripts
#Enter the path to your WordPress installation. Follow this syntax. Note the absence of a trailing '/' (ex: /var/www/html)
WP_PATH=/var/www/wordpress
#Enter the URL of the Nginx statistics page. (ex: http://10.0.0.241/stats)
NGINX_STATS=http://10.0.0.10/viewers
#Enter the path to the HTML assets folder. This is where the server will hold the .m3u files. YOU MUST CHOWN this folder! (ex: /var/www/html/assets)
ASSETS=/var/www/wordpress/assets
#Enter the URL to the assets folder. (ex: http://10.0.0.241/assets)
WEB_ASSETS=http://10.0.0.10/assets
#Enter the path to the recording folder. (ex: /var/www/wordpress/rec/mp4)
REC=/var/www/wordpress/rec/mp4
#Enter the rec path. This is where the recordings will be stored. (ex: http://10.0.0.202/rec/mp4)
REC_PATH=http://10.0.0.10/rec/mp4
#Enter the RTMP server path. (ex: rtmp://10.0.0.242/live)
RTMP_PATH=rtmp://10.0.0.10/live
#Optional: Enter the path to a background image for JWPLayer. (ex: http://10.0.0.241/assets/site_logo.png)
IMAGE_PATH=
#Changes directory to the script-path for execution (required for cron)
cd $SCRIPT_PATH
#Creating the master list file.
MASTER="rtmp_users.list"
/bin/touch $MASTER
/bin/touch toadd.list
/bin/rm toadd.list
/bin/touch toadd.list
#Getting the current statistics from the streaming server
wget -O stats.xml $NGINX_STATS
#Parsing the XML file to get all the handles/keys currently streaming. These should correspond with the usernames of the streamers.
xmlstarlet sel -t -m '//name' -v . -n <stats.xml | grep -v live > unsanitized.list
sed -r 's/^\s*(.*\S)*\s*$/\1/;/^$/d' unsanitized.list > livestreams.list
#The loop below cycles through the livestreams.list file and compares each line to the current master list. If it doesn't exist, it adds it to the master list. It also adds it to the list of things we need to add to the website.
while read NAME
do
if ! /bin/grep -Fxq $NAME $MASTER; then
echo $NAME >> $MASTER
echo $NAME >> toadd.list
fi
done <livestreams.list
#Getting the existing post titles and ID's from WordPress. These will be compared to the currently active streams.
/usr/local/bin/wp --path=$WP_PATH post list --post_type=page --fields=ID,post_title > $SCRIPT_PATH/id_title.list
#Loop to find names to add. If they don't exist, a new page is created for them. This will display at the top of the menu. It is currently empty at this stage. Also checks and makes a blog post that will feature recordings from the user.
while read NEW; do
if ! /bin/grep $NEW id_title.list; then
/usr/local/bin/wp --path=$WP_PATH post create --post_type=page --post_status=publish --post_title=$NEW
/usr/local/bin/wp --path=$WP_PATH post create --post_type=post --post_status=publish --post_title=$NEW" Recordings"
fi
done <$MASTER
#The loop below checks to see if the stream is currently live or dead (depends on if its in the livestreams.list file). If its live, it generates a page with the relavent
#stream information customized for the user. If its dead, it will generate a dead page to notify browsers that the user is not currently streaming. This loop will update
# ALL streams in the master_rtmp.list as often as the cronjob is executed.
while read STREAM; do
echo $STREAM
/bin/ls $REC | grep $STREAM > $STREAM.record.list
cat $STREAM.record.list
POST_ID=`/usr/local/bin/wp --path=$WP_PATH post list --fields=ID,post_title | /bin/grep $STREAM | /usr/bin/awk '{print $1}'`
/bin/echo "<html>" >> $STREAM.record.html
/bin/echo "<head>" >> $STREAM.record.html
/bin/echo "<!--more-->" >> $STREAM.record.html
/bin/echo "</head>" >> $STREAM.record.html
/bin/echo "<body>" >> $STREAM.record.html
/bin/echo "<p>EXPERIMENTAL! Use Chrome or Firefox. Fullscreen for higest quality. If there is nothing here, "$STREAM" doesn't have any recordings. Page is regenerated every 3 minutes</p>" >> $STREAM.record.html
while read LINKS; do
/bin/echo "<a href="$REC_PATH/$LINKS">"$LINKS"</a>" >> $STREAM.record.html
/bin/echo "<video width='100%' controls>" >> $STREAM.record.html
/bin/echo "<source src='$REC_PATH/$LINKS' type='video/mp4'>" >> $STREAM.record.html
/bin/echo "Your browser does not support the HTML5 video player. Please download the file and play locally. Thank you." >> $STREAM.record.html
/bin/echo "</video>" >> $STREAM.record.html
done <$STREAM.record.list
/bin/echo "</body>" >> $STREAM.record.html
/bin/echo "</html>" >> $STREAM.record.html
RECORD_HTML=`cat $STREAM.record.html`
echo "ID BELOW"
echo $POST_ID
/usr/local/bin/wp --path=$WP_PATH post update $POST_ID --post_content="$RECORD_HTML"
ID=`/usr/local/bin/wp --path=$WP_PATH post list --post_type=page --fields=ID,post_title | /bin/grep $STREAM | /usr/bin/awk '{print $1}'`
/bin/rm $STREAM.record.list
if /bin/grep $STREAM livestreams.list; then
echo $STREAM" is online"
/bin/echo $RTMP_PATH/$STREAM > $ASSETS/$STREAM.m3u
/bin/echo "<html>" >> $STREAM.livetemplate.html
/bin/echo "<head>" >> $STREAM.livetemplate.html
/bin/echo "<script src=http://"$IP_ADDRESS"/jwplayer/jwplayer.js></script>" >> $STREAM.livetemplate.html
/bin/echo "</head>" >> $STREAM.livetemplate.html
/bin/echo "<body>" >> $STREAM.livetemplate.html
/bin/echo "<div id="$STREAM"></div>" >> $STREAM.livetemplate.html
/bin/echo "<script>// <![CDATA[" >> $STREAM.livetemplate.html
/bin/echo "jwplayer('"$STREAM"').setup({ file: '"$RTMP_PATH/$STREAM"', image: '//"$IMAGE_PATH"', title: '"$STREAM"', width: '100%', aspectratio: '16:9' });" >> $STREAM.livetemplate.html
/bin/echo "// ]]></script>" >> $STREAM.livetemplate.html
/bin/echo $STREAM" is online and streaming as of "`date` >> $STREAM.livetemplate.html
/bin/echo "<a href="$WEB_ASSETS/$STREAM".m3u>VLC LINK</a> (Right Click and Save-as)" >> $STREAM.livetemplate.html
/bin/echo "<p></p>" >> $STREAM.livetemplate.html
/bin/echo "<h1><a href="http://$IP_ADDRESS"/"$STREAM"-recordings>"$STREAM"'s past recordings</a></h1>" >> $STREAM.livetemplate.html
/bin/echo "</body>" >> $STREAM.livetemplate.html
/bin/echo "</html>" >> $STREAM.livetemplate.html
LIVE_HTML=`cat $STREAM.livetemplate.html`
/usr/local/bin/wp --path=$WP_PATH post update $ID --post_content="$LIVE_HTML"
/bin/rm $STREAM.livetemplate.html
else
/bin/echo $STREAM" is offline"
/bin/echo "<html>" >> $STREAM.deadtemplate.html
#/bin/echo "<head></head>" >> $STREAM.deadtemplate.html
/bin/echo "<h1>"$STREAM" is not streaming as of "`date`" </h1>" >> $STREAM.deadtemplate.html
/bin/echo "<body>" >> $STREAM.deadtemplate.html
#/bin/echo "<p></p>" >> $STREAM.deadtemplate.html
/bin/echo "<h1><a href="http://$IP_ADDRESS"/"$STREAM"-recordings>"$STREAM"'s past recordings</a></h1>" >> $STREAM.deadtemplate.html
/bin/echo "Page is regenerated every 60 seconds, please check again shortly." >> $STREAM.deadtemplate.html
/bin/echo "If this is your stream and it should be running, please contact an event administrator." >> $STREAM.deadtemplate.html #This string can be modified to meet your requirements.
/bin/echo "</body>" >> $STREAM.deadtemplate.html
/bin/echo "</html>" >> $STREAM.deadtemplate.html
DEAD_HTML=`cat $STREAM.deadtemplate.html`
/usr/local/bin/wp --path=$WP_PATH post update $ID --post_content="$DEAD_HTML"
/bin/rm $STREAM.deadtemplate.html
fi
/bin/rm $STREAM.record.html
done <$MASTER
#CLEANUP to remove some temp files that we don't need. These could be converted into arrays at a later point, but it makes it very simple to use files/grep.
/bin/rm stats.xml
/bin/rm id_title.list
/bin/rm livestreams.list
/bin/rm unsanitized.list
/bin/rm toadd.list
#This last line is useful for logging. If you run this script as a cronjob and send the output to a log file, you will see when the last time the script ran.
/bin/echo "Script finished at" `date`
Then we will create the recording_watcher.sh script. This will check for new recordings, and if it finds any, it will process them and move them to the web server directory. From there, users can view them.
sudo nano /opt/nginx_scripts/recording_watcher.sh
#!/bin/bash
#This script watches the /opt/ngnix_rtmp/live_recording directory for .flv files and converts them to .mp4 to make sharing the videos easier within a web browser.
#Enter the path to the script (ex: /opt/nginx_scripts)
SCRIPT_PATH=/opt/nginx_scripts
#Enter the path that NGINX-RTMP is saving files to. (ex: /opt/nginx_scripts/live_recording)
RTMP_RECORD=/opt/nginx_scripts/live_recording
#Enter the path to the webserver directory that the videos will be placed into. (ex: /var/www/wordpress/rec/mp4)
REC=/var/www/wordpress/rec/mp4
cd $RTMP_RECORD
#List all files in the /opt/nginx_rtmp/live_recording directory that match .flv. It writes the list to the flash_vids.list file with the full path.
ls -d -1 $PWD/** | grep .flv > $SCRIPT_PATH/flash_vids.list
#lsof is used to find what files are currently open by the OS. We use it here to see what files are open in the /opt/nginx_rtmp/live_recordings directory that match .flv. If it finds a
#file currently open, it places the name of the file into the writing_vids.list file.
lsof $RTMP_RECORD/* | grep .flv | /usr/bin/awk '{print $9}' > $SCRIPT_PATH/writing_vids.list
cd $SCRIPT_PATH
#Loop below reads through the writing_vids.list file and cross references it to the flash_vids.list file. If there's a match, it removes the active file from the flash_vids.list file.
#These files will not be processed in the next loop.
while read WRITING; do
grep -v $WRITING flash_vids.list > flash.tmp
mv flash.tmp flash_vids.list
done <writing_vids.list
#Sed strips the full path from the file names within the flash_vids.list file.
#!!!!!!!!!!!!!!!!!! ENTER THE CORRECT PATH BELOW. SHOULD BE THE SAME AS RTMP_RECORD VARIABLE. THANK YOU !!!!!!!!!!!!!!!!!!!!!!!!!!!
sed -i 's/\/opt\/nginx_scripts\/live_recording\///g' flash_vids.list
#Loop below reads through the flash_vids.list file and processes each .flv with avconv. It losslessly (and quickly!) converts the .flv video file to .mp4. It then removes the .flv pattern
#from the filenames. At the end it removes the .flv file.
while read FLASH; do
/usr/bin/avconv -i $RTMP_RECORD/$FLASH -vcodec copy -acodec copy $RTMP_RECORD/$FLASH.mp4
/usr/bin/rename 's/.flv//g' $RTMP_RECORD/$FLASH.mp4
/bin/rm $RTMP_RECORD/$FLASH
done <flash_vids.list
#Cleanup. Move all .mp4 files to the /var/www/wordpress/rec/mp4/ directory, and deletes the flash_vids.list and writing_vids.list files.
/bin/mv $RTMP_RECORD/*.mp4 $REC
/bin/rm flash_vids.list
/bin/rm writing_vids.list
Save that file as well. Now you should have 2 scripts, nginx_watcher and recording_watcher. Now we need to assign some final permissions and ownerships, and create a cron job to run every 60 seconds.
sudo chmod a+x nginx_watcher.sh sudo chmod a+x recording_watcher.sh sudo chown www-data:www-data nginx_watcher.sh sudo chown www-data:www-data recording_watcher.sh sudo mkdir /opt/nginx_scripts/live_recording sudo chown www-data:www-data /opt/nginx_scripts/live_recording sudo -u www-data -s #change to the www-data user to add a cronjob. ./nginx_watcher.sh ./recording_watcher.sh #those should both run. crontab -e */1 * * * * /opt/nginx_scripts/nginx_watcher.sh >> /opt/nginx_scripts/nginx_watcher.log */3 * * * * /opt/nginx_scripts/recording_watcher.sh >> /opt/nginx_scripts/recording_watcher.log touch /opt/nginx_scripts/nginx_watcher.log touch /opt/nginx_scripts/recording_watcher.log exit
If all went well, you should now have a fully functional Nginx-RTMP + WordPress installation capable of handling streaming sources and clients. It should be able to record all streams running through it.
Troubleshooting Tips
If you have a large event coming up and would like to use this for your streaming needs, please feel free to email me at coast.willstare [@] gmail.com. I can work with you to provide assistance, or even meet additional requirements.
There may be errors in this guide. If you spot any, please leave a comment. Thanks!
If you have been following these tutorials, you should now have a fully functional Nginx RTMP server as well as a WordPress site optimized for showing live streams.
We just need to install a few more things and this project will be complete!
The first package we need is wp-cli. This program will allow us to modify our WordPress site from the CLI! Cool!
user@websvr:~$ wget https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
Check to see if it works by typing
user@websvr:~$ php wp-cli.phar --info
If it works, we can make it an executable so you just type “wp” to call it.
user@websvr:~$ chmod +x wp-cli.phar user@websvr:~$ sudo mv wp-cli.phar /usr/local/bin/wp
That’s all we have to do to enable wp-cli. Next we need to install xmlstarlet, a utility to parse xml files.
user@websvr:~$ sudo apt-get install xmlstarlet
After that installs, we are finished adding software to the server. Everything else should be included in your distro.
The script we will be using automatically generates .m3u playlist files that will need a directory in the webserver to reside. I like to use an ‘assets’ folder, but you can name it anything you’d like. Make a folder in the www root of your webserver, where WordPress is installed. After we create the folder, we must ‘chown’ it, which means change owner. Change ownership to the user that will be running the script.
user@websvr:~$ sudo mkdir /var/www/html/assets/ user@websvr:~$ sudo chown user /var/www/html/assets/
What follows is a custom script that will allow the WordPress site to update based on the status of the nginx RTMP server. It will require some input before you run it, but once its set it can take care of itself.
#!/bin/bash
#MASTER contains a list of registered usernames for the event. To clear it, simply delete the file!
MASTER="rtmp_users.list"
#Enter the path to this script. Follow this syntax. Note the absence of a trailing '/' (ex: /home/user/scripts)
SCRIPT_PATH=
#Enter the path to your WordPress installation. Follow this syntax. Note the absence of a trailing '/' (ex: /var/www/html)
WP_PATH=
#Enter the URL of the Nginx statistics page. (ex: http://10.0.0.241/stats)
NGINX_STATS=
#Enter the path to the HTML assets folder. This is where the server will hold the .m3u files. YOU MUST CHOWN this folder! (ex: /var/www/html/assets)
ASSETS=
#Enter the URL to the assets folder. (ex: http://10.0.0.241/assets)
WEB_ASSETS=
#Enter the RTMP server path. (ex: rtmp://10.0.0.242/live)
RTMP_PATH=
#Optional: Enter the path to a background image for JWPLayer. (ex: http://10.0.0.241/assets/site_logo.png)
IMAGE_PATH=
#Changes directory to the script-path for execution (required for cron)
cd $SCRIPT_PATH
#Creating the master list file.
MASTER="rtmp_users.list"
/bin/touch $MASTER
/bin/touch toadd.list
/bin/rm toadd.list
/bin/touch toadd.list
#Getting the current statistics from the streaming server
wget -O stats.xml $NGINX_STATS
#Parsing the XML file to get all the handles/keys currently streaming. These should correspond with the usernames of the streamers.
xmlstarlet sel -t -m '//name' -v . -n <stats.xml | grep -v live > livestreams.list
#The loop below cycles through the livestreams.list file and compares each line to the current master list. If it doesn't exist, it adds it to the master list. It also adds it to the list of things we need to add to the website.
while read NAME
do
if ! /bin/grep -Fxq $NAME $MASTER; then
echo $NAME >> $MASTER
echo $NAME >> toadd.list
fi
done <livestreams.list
#Getting the existing post titles and ID's from WordPress. These will be compared to the currently active streams.
/usr/local/bin/wp --path=$WP_PATH post list --post_type=page --fields=ID,post_title > $SCRIPT_PATH/id_title.list
#Loop to find names to add. If they don't exist, a new page is created for them. This will display at the top of the menu. It is currently empty at this stage.
while read NEW; do
if ! /bin/grep $NEW id_title.list; then
/usr/local/bin/wp --path=$WP_PATH post create --post_type=page --post_status=publish --post_title=$NEW
fi
done <$MASTER
#The loop below checks to see if the stream is currently live or dead (depends on if its in the livestreams.list file). If its live, it generates a page with the relavent
#stream information customized for the user. If its dead, it will generate a dead page to notify browsers that the user is not currently streaming. This loop will update
# ALL streams in the master_rtmp.list as often as the cronjob is executed.
while read STREAM; do
ID=`/usr/local/bin/wp --path=$WP_PATH post list --post_type=page --fields=ID,post_title | /bin/grep $STREAM | /usr/bin/awk '{print $1}'`
if /bin/grep $STREAM livestreams.list; then
echo $STREAM" is online"
/bin/echo $RTMP_PATH/$STREAM > $ASSETS/$STREAM.m3u
/bin/echo "<html>" >> $STREAM.livetemplate.html
/bin/echo "<head>" >> $STREAM.livetemplate.html
/bin/echo "<script src=http://jwpsrv.com/library/UjjVOi7iEeSiiiIAC0MJiQ.js></script>" >> $STREAM.livetemplate.html
/bin/echo "</head>" >> $STREAM.livetemplate.html
/bin/echo "<body>" >> $STREAM.livetemplate.html
/bin/echo "<div id="$STREAM"></div>" >> $STREAM.livetemplate.html
/bin/echo "<script>// <![CDATA[" >> $STREAM.livetemplate.html
/bin/echo "jwplayer('"$STREAM"').setup({ file: '"$RTMP_PATH/$STREAM"', image: '//"$IMAGE_PATH"', title: '$NEW', width: '100%', aspectratio: '16:9' });" >> $STREAM.livetemplate.html
/bin/echo "// ]]></script>" >> $STREAM.livetemplate.html
/bin/echo $STREAM" is online and streaming as of "`date` >> $STREAM.livetemplate.html
/bin/echo "<a href="$WEB_ASSETS/$STREAM".m3u>VLC LINK</a> (Right Click and Save-as)" >> $STREAM.livetemplate.html
/bin/echo "</body>" >> $STREAM.livetemplate.html
/bin/echo "</html>" >> $STREAM.livetemplate.html
LIVE_HTML=`cat $STREAM.livetemplate.html`
/usr/local/bin/wp --path=$WP_PATH post update $ID --post_content="$LIVE_HTML"
/bin/rm $STREAM.livetemplate.html
else
/bin/echo $STREAM" is offline"
/bin/echo "<html>" >> $STREAM.deadtemplate.html
/bin/echo "<head></head>" >> $STREAM.deadtemplate.html
/bin/echo "<h1>"$STREAM" is not streaming as of "`date`" </h1>" >> $STREAM.deadtemplate.html
/bin/echo "<body>" >> $STREAM.deadtemplate.html
/bin/echo "Page is regenerated every 60 seconds, please check again shortly." >> $STREAM.deadtemplate.html
/bin/echo "If this is your stream and it should be running, please contact an event administrator." >> $STREAM.deadtemplate.html #This string can be modified to meet your requirements.
/bin/echo "</body>" >> $STREAM.deadtemplate.html
/bin/echo "</html>" >> $STREAM.deadtemplate.html
DEAD_HTML=`cat $STREAM.deadtemplate.html`
/usr/local/bin/wp --path=$WP_PATH post update $ID --post_content="$DEAD_HTML"
/bin/rm $STREAM.deadtemplate.html
fi
done <$MASTER
#CLEANUP to remove some temp files that we don't need. These could be converted into arrays at a later point, but it makes it very simple to use files/grep.
/bin/rm stats.xml
/bin/rm id_title.list
/bin/rm livestreams.list
/bin/rm toadd.list
#This last line is useful for logging. If you run this script as a cronjob and send the output to a log file, you will see when the last time the script ran.
/bin/echo "Script finished at" `date`
Create a folder in the home directory of the user that will be running the script. Inside that folder, create a new file.
user@websvr:~$ nano nginx_watcher.sh
Edit the file
Paste the script into this file, and save it. Make the file executable by typing
user@websvr:~$ chmod +x nginx_watcher.sh
We are almost finished! Start a stream to the Nginx-RTMP server using your streaming software. Then on the webserver, run the script by typing
user@websvr:~$ ./nginx_watcher.sh
If all goes well, there will be no errors reported, and your WordPress site will have a new page set up for the stream. The live video should be playable through JWPlayer, and an m3u file will be available for download. You should also be able to stop the stream, run the script again, and the page will be updated to state the the stream is offline.
The last step is to create a cronjob for this script to run every minute. This will allow users to begin streaming and a page for them will be created no more than 1 minute after.
Type
user@websvr:~$ crontab -e
and create a job run every minute. Here’s mine:
*/1 * * * * /home/user/nginx_water/nginx_watcher.sh >> /home/user/nginx_watcher/log/nginx_watcher.log 2>&1
That will run the script every minute and send the output to a log file.
After that, your Nginx-RTMP/WordPress Frankenstein should be ready to roll!
Screenshot of the webpage when someone is actively streaming.
Screenshot of the webpage when that same person has stopped streaming.
If you have any questsions, please leave a comment below. If you have any suggestions for the script, I’d love to hear them. Thanks for reading.
If you are following along in this mini-series, you should now have a fully functional nginx based RTMP server capable of receiving and broadcasting live video streams. As cool as that is, its not very user friendly, especially for potential viewers.
In this post we will go over the steps to get a frontend set up for our RTMP server. This will serve as a central hub for our streaming environment. It will host information about how to watch the streams, how to broadcast to the server, as well as some cool statistics. You will need the following.
So, let’s start!
Log into your fully up to date WordPress installation via the web browser. I have found a few customization options that make the website look better for streaming.
A basic how-to guide on streaming to the RTMP server with OBS. This page also includes directions for Xsplit as well.
A simple iframe within a page that shows the RTMP stats. Not necessary, but its kind of cool 🙂
A basic how to watch page with download links for VLC.
For this guide I am following this post.
For this guide you will need an up to date Ubuntu Server 14.04 virtual or physical machine.
Update your operating system. If you’re on Ubuntu, its a simple process. Simply type
user@websvr:~$ sudo apt-get update
After that, just type
user@websvr:~$ sudo apt-get upgrade
This will download updates for all the installed software. When that finishes, we need to grab the dependencies for nginx. Type or paste the following into the server.
user@streamsvr:~$ sudo apt-get install build-essential libpcre3 libpcre3-dev libssl-dev
Now that we have the dependencies, we can download the source code of nginx from their webpage. As of this post, the most recent version is 1.7.4. To download it, type the following.
user@streamsvr:~$ wget http://nginx.org/download/nginx-1.7.4.tar.gz
We also need to grab the RTMP module because it isn’t included in the main nginx distribution.
user@streamsvr:~$ wget https://github.com/arut/nginx-rtmp-module/archive/master.zip
Before we continue, we need to install the unzip program.
user@streamsvr:~$ sudo apt-get install unzip
Now we open the archives we downloaded and get cracking!
user@streamsvr:~$ tar -zxvf nginx-1.7.4.tar.gz user@streamsvr:~$ unzip master.zip user@streamsvr:~$ cd nginx-1.7.4/
We can now configure the source and build the binary.
user@streamsvr:~/nginx-1.7.4$ ./configure --with-http_ssl_module --add-module=../nginx-rtmp-module-master user@streamsvr:~/nginx-1.7.4$ make user@streamsvr:~/nginx-1.7.4$ sudo make install
Now nginx is installed! It is install ed in /usr/local/nginx. To start it type
user@streamsvr:~$ sudo /usr/local/nginx/sbin/nginx
Test it by going to your server IP with your web browser. It should display the following:
We are almost done. To add RTMP functionality, we must enable it in the nginx configuration file.
user@streamsvr:~$ sudo nano /usr/local/nginx/conf/nginx.conf
Paste this in at the end of the file.
rtmp {
server {
listen 1935;
chunk_size 4096;
application live {
live on;
record off;
}
}
}
To save, hit Crtl+x , yes, and enter.
To make the rest of the project easier, we should make nginx into a system service that runs automatically and can be controlled with the service command.
sudo nano /etc/init/nginx.conf
Paste this into the file
# nginx description "nginx http daemon" author "Philipp Klose <me@'thisdomain'.de>" start on (filesystem and net-device-up IFACE=lo) stop on runlevel [!2345] env DAEMON=/usr/local/nginx/sbin/nginx env PID=/var/run/nginx.pid expect fork respawn respawn limit 10 5 #oom never pre-start script $DAEMON -t if [ $? -ne 0 ] then exit $? fi end script exec $DAEMON
We can now start and stop nginx with the sudo service nginx start | stop | restart commands. Yes! It will also start when the server boots, so you don’t need to worry about that.
Next we need to go back to the nginx configuration file and prepare it for stats delivery. The ability to display server statistics will be crucial to this project. This page will ensure that live streams have the correct pages generated for them.
Go back to the nginx configuration and add the following inside the server directive in the /usr/local/nginx/conf/nginx.conf.
# rtmp statistics
location /viewers {
rtmp_stat all;
rtmp_stat_stylesheet stat.xsl;
}
location /stat.xsl {
# you can move stat.xsl to a different location
# under linux you could use /var/user/www for example
root html;
}
The server directive should now start like this:
server {
listen 80;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
#rtmp statistics
location /viewers {
rtmp_stat all;
rtmp_stat_stylesheet stat.xsl;
}
location /stat.xsl {
# you can move stat.xsl to a different location
# under linux you could use /var/user/www for example
root html;
}
#error_page 404 /404.html;
We have to do one more quick thing before the stats page will work. In order to generate the XML, the RTMP module uses an XSL file. This will need to be placed in the HTML root of the nginx server. Create a file called stat.xsl containing the following. Save it, and we should be good to go. For whatever reason, pasting the stat.xsl file into this page is messing things up, so I’ll just include a link to the file.
user@streamsvr:~$ sudo nano /usr/local/nginx/html/stat.xsl
After you paste that in, save the nginx configuration. Restart the entire server. After it boots should be able to see the rtmp server stats if you go to your server IP address /viewers.
And that’s it for the nginx-rtmp installation/configuration. The server is now running and ready for the next section of this series… Web server configuration and scripting!
I am starting a mini-series of posts on how to set up Nginx-RTMP and WordPress to create a local version of Twitch.tv. I will post detailed guides on installing and configuring all software required for this project.
Project requirements are as follows:
The end product will be a fully functional streaming server capable of handling 1gbps of video traffic, with a custom WordPress frontend for users to use to access the streams. The entire system is automated and requires minimal user input. A basic use-case is below.
Bob wants to stream his Dota game for the entire LAN party to see. The LAN has a limited bandwidth internet connection, so using a service like Twitch.tv is out of the question. However, the LAN execs have configured a local RTMP solution for him to use.
Bob configures his streaming software like he would normally, except he points his stream to the local RTMP server. He uses his gamer handle (b0b) as a stream key. At this point, Bob’s job is finished. He can start playing his game.
On the backend, a script reads that a new stream has started. It checks to see if this stream has been played before, or if this is a new stream all together. If its new, it adds the stream to the local database and creates a page on the WordPress site dedicated to Bob’s stream. It then runs through every stream that has been registered to the server during the event. If the stream is active, like Bob’s, it updates the page with a JWPlayer configured specifically for Bob’s stream. It also generates an .m3u file with a playlist entry pointing towards Bob’s stream mount point. Viewers can just navigate to Bob’s page on the WordPress site and start watching from within their web browser.
If some of the previously registered streams are offline, it updates their page with an offline message telling users to check back soon. By using this process, every single page is updated every 60 seconds via a cronjob.
So that’s a basic overview of what this setup can do. It can listen for, create, and distribute live RTMP streams to viewers automatically and painlessly. And because all traffic is local, no internet bandwidth is wasted for redundant connections to websites like Twitch.tv or hitbox.
I will be making many more posts about how to set this up for your own event, so stay tuned. If you have any questions or comments, please leave a message below.
Part 1 – Install nginx rtmp tutorial here!
Part 2 – Customize WordPress tutorial here!
Part 3 – Finishing the RTMP WordPress Project here!
Part 4 – Virtual Machine Download here!