How to count traffic in Linux. Review of Free Billing Systems on Linux Logging Activity Using iptables

14

I would recommend iptraf or iftop if you don't need a lot of functionality. On the iptraf home page:

IPTraf is a console network statistics utility for Linux. It collects many metrics such as TCP connection and byte counts, interface statistics and activity indicators, TCP/UDP traffic failures, and LAN packet and byte counts. Characteristics

• An IP traffic monitor that shows information about the IP traffic passing through your network. Includes TCP flag information, packet and byte counts, ICMP data, OSPF packet types.
• General and detailed interface statistics showing IP, TCP, UDP, ICMP, non-IP and other IP packets, IP checksum errors, interface activity, packet count.
• TCP and UDP service monitor showing incoming and outgoing packet counters for common TCP and UDP application ports
• LAN Statistics module that detects active hosts and displays statistics showing data activity on them
• TCP, UDP, and other protocol display filters that allow you to view only the traffic you are interested in.
• Registration
• Supports interface types Ethernet, FDDI, ISDN, SLIP, PPP and loopback.
• Uses the built-in raw socket interface of the Linux kernel, allowing it to be used on a wide range of supported network cards.
• Full screen menu driven mode.

Screenshot of IPTraf main menu:

This is a screenshot if iftop:

1

Things like bro IDS will analyze the traffic that goes through the network interface and record all sorts of things like connections and amount of IR traffic, protocols found and information for each protocol (eg HTTP requests, emails sent, DNS queries, general SSL certificate names...). This won't tell you what kind of application it is (except for registering user agents, such as for HTTP browsers). Because it sniffs the packet, it may miss some data if it can't keep up with the volume of data exchanged (though it will report if it does).

conntrackd can be used to log each connection tracked by a stateful firewall and exchange data. It will work regardless of the amount of data passing through the system, but will not report data that does not pass through the firewall, such as bridged traffic, if it is excluded from the network filter or raw socket.

You can also use firewall rules to log traffic using the LOG or ULOG target in combination with ulogd .

You need to use the auditing system (auditd/auditctl) to log what's related to the pid connection, but that would be very large and not easy to parse.

ALEKSANDR KUZNETSOV

Many of you in your work are faced with the need to keep statistics of traffic consumed in the local network. We will tell you how this task is solved in Linux without any financial investment in specialized software products.

We will take into account the traffic passing through the LAN gateway to the Internet using ipcad (Cisco IP accounting simulator). This is a program that can count using several mechanisms, such as the BPF, libpcap and iptables ULOG interfaces. The material is intended for those administrators in charge of a small or medium local network. All that is needed for work is just ipcad itself, a text editor for writing scripts for parsing statistics, and a great desire to finally solve the problem of calculating the consumed traffic in the local network.

We will look at the processes of installing, configuring and running ipcad, as well as some examples of writing scripts for parsing statistics.

Installing ipcad

First you need to download the latest version of ipcad (http://lionet.info/ipcad). At the time of writing this article it was 3.6.6. There is nothing nontrivial in the installation process itself:

# tar -xvzf ipcad-3.6.6.tar.gz

# cd ipcad-3.6.6

# ./configure

# make

# /bin/su -

# make install

In RPM-based distributions, it is better to install ipcad using checkinstall (in order to follow the rules of the package system).

Setting up ipcad

Of the many ways to maintain statistics, the simplest is to take into account the traffic that passed through the physical interfaces of the gateway (in this article, two Ethernet interfaces). To specify the description of the settings, we took a local network with one subnet 192.168.0.0/24.

By default, the only ipcad configuration file is located in the /usr/local/etc directory. For the task of accounting for traffic through two network interfaces eth0 and eth1, ipcad.conf looks like this:

# /usr/local/etc/ipcad.conf

# GLOBAL OPTIONS

# The capture-ports option enables/disables additional fields in statistics, such as: TCP and UDP ports, as well as

# ICMP packet types. However, enabling this option increases memory consumption and reduces calculation speed

# traffic and in some cases distorts the output via RSH, so it is disabled in this example

capture-ports disable;

# The size of the buffers used to transfer statistics by the kernel is 64 KB by default, which is quite enough

buffers = 64k;

# INTERFACE OPTIONS

# Interfaces on which passing traffic is counted.

# Consider the case of a gateway with two network interfaces: internal (local network) and external (Internet)

interface eth1; # ...and one more

# Separate statistics per IP address for subnet 192.168.0.0/24. "aggregate 192.168.0.0/24" indicates

# ipcad - network address range. "strip 32" means that all 32 bits must be entered into the statistics

# address belonging to this address range

aggregate 192.168.0.0/24 strip 32;

# RSH SERVER OPTIONS

# Settings for the rsh server that will be used to view statistics.

rsh enable at 127.0.0.1;

# The rules below describe the policies for accessing ipcad statistics. Root can fully control

# (make a backup, view and change the calculation tables). Everyone else can only view statistics

rsh [email protected] admin;

rsh [email protected] backup;

rsh [email protected];

rsh 127.0.0.1 view-only;

# Lifetime and IP Packet Timeout

rsh ttl = 3;

rsh timeout = 30;

# The dumpfile option specifies the path to the file into which statistics data will be stored by default

dumpfile = /var/log/ipcad/ipcad-curr.dump;

# OTHER OPTIONS

pidfile = /var/run/ipcad.pid;

# The memory_limit option specifies the amount of memory to store the contents of one data stream.

# The syntax is: memory_limit =<количество>[(k|m|e)] ;

# where k - KB; m - MB; e – number of data table rows

memory_limit = 10m;

In our case, all other options specified by ipcad.conf can be safely removed (or at least commented out) as unnecessary. This way they won't be involved at all.

Note: the /var/log/ipcad/ directory is not created when installing ipcad, so you need to create it yourself. I recommend giving rights to read, write and view the contents of the directory with ipcad logs only to the root user. For everyone else, block everything to prevent even ordinary users from viewing statistics of traffic consumption on the network.

At this point, the installation and configuration processes are completed, and you can proceed to launching ipcad.

Launching ipcad

It makes sense to run ipcad in such a way that when loading, it restores statistics data from the dumpfile, goes into the background, and when turned off, resets all data to the same dumpfile. The command to run looks like this:

/usr/local/bin/ipcad –rds

Where:

• key r– imports data from dumpfile;
• key d– “demonizes” ipcad;
• key s– resets statistics to dumpfile when ipcad is turned off.

It is best to place the specified command in one of the init scripts to launch ipcad along with the system. I am not specifying which init script, because... it depends on each distribution. For example, in SuSE Linux the script /etc/init.d/boot.local is used for similar purposes, and in RedHat - /etc/rc.d/rc.local.

View statistics

To manage ipcad statistics, rsh is used, the settings of which are specified in the ipcad.conf configuration file (as discussed in “Configuring ipcad”).

The general command syntax for ipcad is as follows:

rsh host command

where host is the host on which statistics are kept, and comand is the command itself. In this case, the host value is localhost.

By command:

rsh localhost help

A complete list of commands is available.

Namely:

• show ip accounting– show statistics.
• clear ip accounting– reset statistics to the checkpoint. If the checkpoint is not specified, then the statistics are reset to zero.
• show ip accounting checkpoint– show statistics saved in checkpoints.
• clear ip accounting checkpoint– reset all control points.
• show ip cache flow– show NetFlow cache.
• show interface – show interface counter .
• dump [ ] – save current statistics to a file If If not specified, the statistics will be reset to the dumpfile specified in the ipcad.conf configuration file.
• restore [ ] – restore statistics.
• import [ ] – import (add) statistics.
• stat– show the current operating status of ipcad.
• show version– show version and uptime ipcad.
• shutdown– shut down ipcad.

To view statistics, just:

rsh localhost show ip accounting

However, in this way information about all traffic passing through the gateway will be displayed on the screen. That is, one table will contain statistics on all incoming and outgoing traffic from the local network.

You can obtain the necessary data from the output of the “show ip accounting” command, for example, using scripts. Which you can write yourself for each specific situation. Below is an example to view statistics for one specified IP address and calculate the total traffic for that IP:

traff.acc

#!/bin/bash

# name of script: traff.acc

HOST=localhost

rsh $HOST dump

rsh $HOST show ip accounting | grep -E "192\.168\.0\."$1"([^0-9]|$)"

echo

echo "Summary traffic of 192.168.0.$1 (kbytes):"

rsh $HOST show ip accounting | grep -E "192\.168\.0\."$1"([^0-9]|$)" | awk "(s+=$4) END (print(s/1024))"

Easy to use:

./traff.acc ip

where ip is a number between 1 and 254 (simply the last octet of the IP address).

The result of this script will be to display the amount (in megabytes) of incoming traffic for the entered IP address.

The simplest script given will display all the summary statistics for the specified IP address, collected from the moment ipcad was first launched until the current point in time. This is not always convenient, since it is more clear to have data for a specific period of time (day/week/month).

It is enough for me to keep statistics of consumed traffic by month. The scheme of such accounting is as follows: at 23 hours 59 minutes. On the “last” day of each month, statistics are dumped into a file, to the name of which it is advisable to add the values ​​of the year and the current month. Just a minute later, at 0:00 a.m. On the 1st day of the next month, the traffic counter is reset with the “clear ip accounting” command, and statistics begin to be kept “from scratch”. Upon reaching the “last” day of the month, the pattern is repeated.

To implement this mechanism, a file /etc/cron.d/ipcad was created with tasks for cron:

/etc/cron.d/ipcad

* * * * * root rsh localhost dump >/dev/null

59 23 * * * root date "+\%Y\%m" > /var/log/ipcad/ipcad.date

0 0 1 * * root (rsh localhost dump /var/log/ipcad/ipcad-$(cat /var/log/ipcad/ipcad.date).dump && rsh localhost clear ip accounting) >/dev/null 2>&1

The first task from the above file dumps statistics every minute. This is done to protect against unexpected shutdowns and freezes of the gateway, which records the passing traffic.

The second line is such a small “hack” that the author of the article had to do due to the difficulty of specifying the “last day of the month” in the schedule when setting up cron jobs. The bottom line is that every day the current month and year (in YYYYMM format) is stored in the /var/log/ipcad/ipcad.date file. The same procedure will take place on the last day of the month. In a minute, the third task will dump the statistics to a file with the contents of the ipcad.date file attached to its name (i.e. the year and the past month in the YYYYMM format), and reset the counter of consumed traffic. On the same day, the ipcad.date file will be overwritten with the new values ​​of the current month and year. It's not exactly pretty, but it works.

Now that we have traffic data for each month, located in different dump files, we can modify the script to add the ability to view general and private traffic statistics for each IP address for a particular month:

traff.acc.2

#!/bin/bash

HOST=localhost

USERS=/etc/hosts

case $1 in

M|m) echo "Summary LAN traffic of month $2 (Mbytes):"

Rsh $HOST dump

Awk "(s+=$4) END (print(s/1048576))"

< /var/log/ipcad/ipcad-$2.dump

Exit 0;;

U|u) grep -E "192\.168\.0\."$3"([^0-9]|$)" $USERS | awk "(print($2))"

USER=`grep -E "192\.168\.0\."$3"([^0-9]|$)" $USERS | awk "(print($2))"`

Rsh $HOST dump

Cat /var/log/ipcad/ipcad-$2.dump | grep -E "192\.168\.0\."$3"([^0-9]|$)"

Echo

Echo "Summary traffic of $USER in $2 month (kbytes):"

Cat /var/log/ipcad/ipcad-$2.dump | grep -E "192\.168\.0\."$3"([^0-9]|$)" | awk "(s+=$4) END (print(s/1024))"

Exit 0;;

*) echo "Usage: "

Echo "1) if only month stat:"

Echo " ./traff.acc.2 m "

Echo" - number of month"

Echo "2) if user’s stat for month"

Echo "./traff.acc.2 u "

Echo" - stat for that month"

Echo" - last octet of user’s IP address"

Echo "note: for current month in flag just type "curr""

Exit 0;;

esac

In the above traff.acc.2 script, the $USERS variable defines a file that contains correspondence between user names and their IP addresses (in this example, this is /etc/hosts on the gateway). This variable is optional and is entered into the script solely for clarity of the statistics being viewed.

You can use the above script in two ways:

./traff.acc.2 m

where semaphore m indicates that general statistics for the specified month should be displayed.

./traff.acc.2 u

The semaphore u means that it is necessary to display the statistics of the specified user for a certain month.

The flag is a six-digit month number in the YYYYMM format - this format for storing monthly dump files is selected in the cron task file (to display statistics for the current month, you should specify the word “curr” instead of the month number).

The flag is the value of the last octet of the IP address of the user whose statistics should be viewed.

In conclusion of the article, I would like to note that instead of a shell script, you can write a script in Perl and, adding navigation elements, attach the page with this script to the httpd server. Then the statistics will be available for viewing through a browser.

I decided to update the blog on 03/21/10

A lot has already changed in 2 years. But normal billing is still not enough. Traffpro (formerly Office-Control) emerged as a special leader. I chose this one; the free version is quite enough for office management.

I will leave the review itself unchanged, as part of history. Most of the information in it is still relevant. Enjoy reading:)

It was necessary to install a traffic accounting and control system at work. As always, I set about searching for a Free system for OC Linux.

What I wanted to find for myself:
1) Accounting for everything (for all ports)
2) Detailed statistics of who is going where and why
3) Authorization of users (so that no one else except the user himself could pump up traffic)
3) So that the user can see how much he downloaded
4) Minimum intervention in the OSWin user.
5) Easy to install and configure billing

Looking ahead a little, I would like to say that I found such a system, and now it is successfully working at the enterprise, and in this review I will consider this assembly.

Friend google.com gave me a lot of interesting and useful information, but as it turned out, there are not so many normal and free billing systems.
They are all divided into 2 types, each with its own pros and cons:
1) On Iptables -
+ Count traffic on all ports and in all directions. = Counting accuracy
+ Fixed by IP and sometimes by MAS address.
+ Stable operation under heavy loads.
+ Good system of Tariffing and disconnecting users
- Not flexible, accounting only by computers, not by users
- No statistics on pages visited
- Often, there is no authorization for users.

2) On Squid -
+ Detailed statistics about the sites visited by each user
+ Ability to block certain sites and install a banner cutter
+ Developed account system (users can either be tied to a computer address or be completely free, i.e. access the Internet using their username and password from any computer)
+ Various authorization methods (ip, ntlm, ncsa - if it’s not clear what this is, then below I’ll tell you what’s what :))
+ Ability to limit traffic.
- traffic is counted only on port 80!
- From this it follows that we will not be able to control mail, ftp, bank clients and similar programs!
- You will have to register the Proxy Address on client machines to use login and password authorization.

I would like to note that some billing systems do not have any of the disadvantages, but they also have their “cockroaches” :)

Now let's look at the candidates themselves.

Billings on iptables:

NeTAMS (Network Traffic Accounting and Monitoring Software)

www.netams.com

Runs under Unix (Linux/FreeBSD/Solaris). Various methods of collecting statistics (tee/divert/ip_queue/ulog/libpcap/netflow v5 and v9/netgraph), storing in a database (BerkleyDB/MySQL/PostgresSQL/Oracle/Radius), aggregation, display, notification, etc. are supported. blocking based on quotas, authorization, balance exhaustion (billing); manage bandwidth, control MAC address spoofing, communicate with RADIUS, create flexible accounting and filtering policies. Read more about the possibilities in this list.

Many admins praise it, but I don’t know what they found in it, maybe they like to dig through a bunch of program code to find the necessary dependencies. I tried to install it on AltLinux 2.4, Ubuntu, Fedore. It didn’t start normally on any system, something was missing in the dependencies, but after reading in detail in the manuals, I realized that even if it worked, I would have to spend more than one night and more than one week to bring it into divine shape.

Didn't work!

Conclusion:
- Only for pros.
- Confusing documentation
- You need to finish a lot yourself
- there are no sh, rpm and .deb packages

STARGAZER

The StarGazer system is designed for authorization and accounting of traffic in local, home and office networks. When creating this system, the task was set to create a product that would meet the requirements of most local networks to account for traffic and client funds, as well as secure client authorization.

Main features of the system:

• control over network clients, their addition, removal, current adjustments
• client authorization, followed by permission or denial of access to the Internet
• counting traffic according to predefined directions and rules
• counting of funds spent by the client and automatic shutdown if they are completely spent
• storing additional information about the client, such as home address, phone number, etc.
• automatic ping of all network clients and displaying the results on the screen
• maintaining a credit history for all clients
• prompt provision of information to the client about his traffic and availability of funds
• generation of detailed reports on the status of clients
• external modules

More details at http://stargazer.dp.ua/doc20/about.html

Administrators say that this is an excellent (almost the only) option for those who want to get maximum functionality at minimal cost. The main focus of billing is for small Internet providers.
I really liked the functionality and billing capabilities. And finally, Detailed and Comprehensive instructions for installation and use!!! (unfortunately, all free systems suffer from the lack of proper documentation, either the developers are too lazy to write it, or they think that we ourselves will figure out how to install it). But I didn’t have time to rejoice for long.
I put everything on the Ubuntu system.
Immediately during the build, strange problems arose in the build script itself (./build). It turned out that the developer wrote make instead of gmake (later I find out that this is for compatibility with FreeBSD). Okay, we dealt with these, but then there was problem after problem - the impossibility of compiling on certain gc++ packages, the traffic control daemon not starting, the onConnect/onDisconnect scripts not working.
Next, I try to build this thing on an AltLinux 2.4 system from a ready-made package, but I also encounter a dozen problems there.
After about a week of torment trying to get it to work normally, I abandoned this matter, especially since I learned from forum members that this billing sometimes loses information from its own database and for security it needs to be transferred to Mysql, and this is another additional hassle. And the developer’s attempts to “please” both Linux (all distribution kits at once!) and FreeBSD did not lead to anything good (as they say, if you chase two birds with one stone, you won’t catch a single one). I don’t know how it is on BSD, but on Linux this billing

Didn't work!

Conclusion:
+ Good website with detailed manual
+ Simple control system in the GUI (windowed) shell
+ Thoughtful client authorizer
- Installation complexity due to the desire to please both BSD and Linux
- No rpm and .deb build packages (and sh is designed in such a way that it only complicates the installation)

NiTraf

http://nitalaut.sarkor.uz/nitraf-uchyot-trafika-v-linux/

What can NiTraf ?
At the moment the list of possibilities is as follows:

Traffic counting per day/since the beginning of the month/since the beginning of the year, broken down by IP address

Ability to receive reports for an arbitrary day/month/year broken down by IP address

Ability to receive reports for an arbitrary day/month/year at a specific IP address

Possibility of setting quotas for traffic (separately for incoming and outgoing).
If the limit is exceeded, the user is disconnected from the Internet (a corresponding rule is added to iptables)

For ease of viewing reports, it is possible to assign aliases (pseudonyms) to IP addresses, under which they will appear in reports

For each IP address, you can view detailed statistics (broken down by port and protocol). The list of ports can be changed

More details at http://nitalaut.sarkor.uz/nitraf-uchyot-trafika-v-linux/

The simplest billing, everything is imaginary. Made money on Ubuntu system. I haven’t tried it on others, as it works fine on Ubuntu

WORKS!

Conclusion:
+ Detailed installation instructions for Debian systems http://nitalaut.sarkor.uz/instrukcii/
+ Speed ​​of system deployment
+ Detailed Web statistics
+ Possibility of accounting for various ports
- Control over IP only (the user can change his IP and use someone else’s traffic)
- No login and password authorization for users

Office-Control

The "Office-Control" system is designed to manage and protect office networks. It is developed taking into account the need for projects that eliminate tedious server setup, and allows you to minimize the time for system deployment and maintenance. It includes several components from which you can build the necessary complex for any office needs.

The complex allows:

• control access to the server both from the Internet and from the internal network
• prevent common attacks on the server using an intelligent protection system
• manage access from the internal network to the external one, and from external to the internal one
• keep a log of all unauthorized attempts to access the server and the secure internal network
• send email notifications to complex administrators in case of attempts to hack the system or in critical situations
• control traffic consumption by internal network users
• restrict user access to external network resources by ports and addresses

The system provides reports on traffic consumption by users, contains an administrator terminal for two operating systems Windows and Linux, and also has a Web client module that is platform independent. There is also a traffic indicator for the network user, which allows the user to receive reports without administrator intervention.

The project is now actively developing and is just leaving the beta testing stage. But now I have quite successfully installed this system. And the developers are actively continuing to develop the project. For example, as soon as I mentioned that it didn’t hurt to do authorization in billing via the web interface, 2 days later this module appeared on the website! The only thing that may be difficult is the documentation, while it is crude, but it is completely replaced by consultation on ICQ, the developer not only helped with installation, but even advised on the work of billing with Iptables and helped set up security rules (in the future they plan to transfer firewall management functions in the web admin panel in order to completely simplify the administrator’s work). Let's see what they can come up with :)
And I forgot to mention, billing is tailored for RedHat Systems (Fedora, CentOS, RedHat)
Personally, I have it on Fedore 8

WORKS!

Conclusion:

+ Good project support
+ Binding not only by IP but also by mac!
+ Detailed Web statistics
+ Authorization by login and password via the Web interface
+ There is an rpm package
+ Stable work with Squid
- Dampness (at the time of writing the article, the web admin panel was not yet ready; we had to add users through scripts.)
- all other minor disadvantages are more likely to be related to dampness than to deep-rooted shortcomings. I will be closely monitoring the further development of the project...

SUB Billing

http://www.subbilling.ru/

About SUB Billing:
This is a system for collecting information about user traffic through ulog-acctd. The processed data is written to a MySQL or PostgreSQL database. A convenient, simple and intuitive web interface is used to manage billing.

Characteristics:
+Data collection - ulogd
+Storage databases - mysql, postgresql
+Authorization - freeradius, chap-secrets
+ Scripts launched by cron

Possibilities:
+Work through - NAT, PPTPD (VPN)
+User groups: each group can be assigned a traffic discount.
+Tariff plans with a billing period, subscription fee, installation of prepaid traffic.
+Setting tariff zones.
+Setting time zones.
+Setting holidays.

+Payment cards.
+Technical support.
+Credit an arbitrary amount to the account.
+Administrator groups with different allowed actions.
+Detailed traffic

In the web user interface:
+Detailed account information
+ A detailed report on payments, connections, traffic in general, and for different tariff zones.
+Payment by express card
+Settings of trusted addresses, password, change of tariff plan.
+Sending requests to tech. support
+News on the main page

I tried to install it on Debian, but something didn’t work in the settings VPN. I didn’t understand it for a long time, since in order to work with billing on each client machine, you need to create VPN connections, and this did not fit into my conditions for ideal billing. This system is mainly used by Gentoo users, although the manuals are for Debian. And yet on Ubuntu

Didn't work!

Conclusion:
+ A good site, although I had to look for the manual (it’s in the archive with billing)
+ According to the screenshots, a good web admin
+ Developing project
- Problems installing on standard Debian
- No rpm and .deb build packages

Billings for SQUID:

Sarg + Wemin

I assembled this system on Ubuntu
The instruction is simple to mess - Go to www.webmin.com and download the latest version of the Debian Package.
And then sudo apt-get install squid sarg(if it doesn’t work, then you are here forum.ubuntu.ru)
Further in webmine we find the contribution Squid (something there) report(oh yes, you will have to tinker with squid a little more, but it's not so difficult, because you can do all the settings through webmin)

This system worked for me successfully in tandem with NiTraf on Ubuntu 7.4

WORKS!

+ Easy to install (there is a lot of information on the Internet)
+ You can freely view from which IP which sites were viewed, how many were downloaded from these sites and the like.
- Users themselves cannot view statistics only for themselves
- You have to generate reports manually

SQUID Account Management System(SAMS)

http://sams.nixdev.net/
http://sams.perm.ru/
Definitely the leader among all billing systems on Squid!
Unfortunately, on Ubuntu 7.4, it didn’t want to work for me in any way, but for some reason it worked great on Fedore 8. I liked the convenient Web admin panel, the wide possibilities for distributing user rights (limited traffic, bandwidth, sites).

WORKS!

+ Detailed installation instructions on the developer's website
+ Excellent web admin
+ Extensive user rights management options
+ Users have their own personal account, where they can see their statistics (MB and visited pages)
+ You can freely view from which IP which sites were viewed, how much was downloaded from these sites, and the like.
+ Well-thought-out system of interaction with Squid
- I would improve the admin panel design a little more
- Counts traffic only on port 80

Now my choice:
Linux Fedore 8
Office-Control + SAMS (+ transparent proxying)

I combined 2 types of billing to get:

Office-Control:
1) Traffic accounting on all ports
2) user authorization via the web interface
3) managing access rights for users and traffic limits
SAMS:
4) Detailed statistics on visited sites for both the administrator and the user
5) saving traffic due to traffic caching
6) Restriction on visiting certain sites

In the next article I will tell you how to assemble, configure and run all this together. I will give examples of working configs and my own builds.

P.s.
For developers and fans who have not earned billings- During testing, I had neither time nor knowledge. On average, I experimented with billing for 5 days; if this was not enough to launch it, then I took on another one. I will be glad if you indicate the reasons why your billing did not start or give detailed, really working installation instructions! (my contacts are at the very end)

P.p.s. (21.10.08)
I haven’t updated this article for a long time, but I still receive requests to me on this issue. I’ll answer this right away. Now my choice is TrffPro.ru(formerly OfficeControl). It has everything you need, and combines the capabilities of SQUID billing (Sams) and Tablov.

© Copyright 2007 Lek (ICQ 355767)

For those users who are deprived of the opportunity to use an unlimited tariff plan (for example, they are forced to often use 3G while roaming), accounting for traffic consumption is a rather critical task. How to solve it in Ubuntu? Let's look at 2 interesting applications.

Network Traffic Monitor (NTM)

This application displays a fairly detailed report on the use of the specified Internet connection. This report includes:

1. current download-upload speed
2. total amount of traffic, amount of traffic for the current ppp session, amount of traffic per month
3. time of the current ppp session (relevant when paying for the Internet is hourly, I don’t know where else this exists)

The program allows you to specify the interface that is being monitored and, if it is a PPPoE connection, provides the ability to automatically terminate the connection when the traffic/time limit specified in the settings is reached for a specified period of time (day/week/month or any other interval). Unfortunately, despite having such rich capabilities, NTM is rather poorly integrated into Ubuntu, everything is limited to a static indicator, and to see the current situation with limits you have to open the program window, which is not always convenient.

Download Monitor

The program was created as part of the Ubuntu Apps Showdown. Its capabilities are much more modest: download monitor allows you to set daily, weekly and monthly traffic limits, receive messages about exceeding these limits and view simple graphs. On the other hand, the program is quite well integrated into Ubuntu; there is a progress bar on the Unity icon, which allows you to estimate how much traffic is left.

You can install the program using the ppa repository using the following commands:

Sudo add-apt-repository ppa:duncanjdavis/download-monitor-submit sudo apt-get update sudo apt-get install download-monitor

How do you control your Internet consumption?

This article contains ways to monitor traffic passing through a server. Most often, the monitoring task arises for routers that provide communication with the Internet.

Real-time monitoring

To monitor traffic in real time, you can use the jnettop utility:

sudo jnettop

Or, as an option, vnstat with the –live switch:

sudo vnstat --live

You can also use a fairly convenient utility with a pseudo-graphical interface called iptraf:

sudo iptraf

Logging activity using iptables

The standard iptables packet filter can sometimes be very useful for analyzing a specific type of traffic. iptables allows you to record activity based on selected criteria in the system log. For example, using iptables you can easily track which computer is sending data to an external network via port 25 (SMTP), thus identifying the infected computer on the local network.

The LOG (and more powerful ULOG) action is used to write data to a log file. Its detailed syntax and possible parameters can be found in the iptables documentation.

An important aspect is that iptables itself does not maintain log files. Instead, the LOG action sends information to the system logging daemon, which then writes it to a log, which is most often /var/log/syslog. The end result is a mess that is very difficult to sort out. Since iptables itself does not do logging, it is impossible to redirect a record to a separate file using iptables itself.

Luckily, Ubuntu has an easy way to sort your data into the right logs using the rsyslog logging system. It is enough to add a unique prefix when writing to the log. For example, like this:

sudo iptables -t filter -A FORWARD -s 192.168.0.0/ 16 -m tcp -p tcp --dport 25 -j LOG --log-prefix "iptables: "

Now all log entries that match this criterion will start with "iptables: ". All that remains is to add a sorting rule to the logging system. To do this, you can create a file /etc/rsyslog.d/10-iptables.conf with the following content:

:msg, contains, "iptables: " -/var/log/iptables.log & ~

The second line means that there is no need to do any further work with this record, i.e. it should not end up in other journals.

After creating a new configuration file, don't forget to reconfigure rsyslog:

sudo /etc/init.d/rsyslog reload

Now all activity matching the added iptables rule will be logged to the /var/log/iptables.log file. If you want to leave logging for a long time, then do not forget to add this file to the log rotation system logrotate, otherwise at some point you may run out of disk space.

You can read more about managing log entries in the rsyslog documentation.

Vnstat and Vnstati utilities

Vnstat is a very simple traffic tracking tool. It is available in the Ubuntu repositories, so installation is straightforward. Vnstat does not require fine-tuning; it simply counts traffic and allows you to view statistics. For the average user, this is often more than enough.

To start counting traffic, you need to run a command that will create a database for the specified interface:

sudo vnstat -u -i eth0

Traffic is counted automatically, and the database is updated when the network device is turned off. But it is possible to update the database manually:

sudo vnstat -u

To view statistics, simply enter:

You can also use the following keys:

H, --hours by hour -d, --days per day -m, --months per month -w, --weeks per week -t, --top10 top10 -s, --short short output -l, - -live monitoring in real time

Vnstati is a utility for visualizing statistics, everything is also simple with it. Here, for example, is a short script that displays statistics on the screen in the form of a picture:

#! /bin/bash vnstat -u vnstati -vs -o ~/ vnstati.png eog ~/ vnstati.png

Both utilities have some other features, don't forget about the --help switch and the man utility!

Netflow - monitoring with statistics

For monitoring with statistics, one of the popular solutions is Netflow technology.

To collect statistics using Netflow, you need to attach special Netflow sensors to the required interfaces, which will collect information and transmit it to the Netflow collector, which can be located on another machine.

The information collected by the collector can be visualized using a graphical frontend, or analyzed using command line utilities.

You can use fprobe or softflowd available in standard repositories as sensors.

fprobe sensor

To install the sensor fprobe For example, you can run the following command:

sudo apt-get install fprobe

The installer will ask on which interface to listen for traffic, and where to transfer the collected statistics - enter the parameters you need and fprobe will start working. For example, for Nfsen in the example below you need to specify port 9995.

Autostart on network restart

The fprobe daemon always crashes when the interface it is bound to goes DOWN. And does not automatically start again when the interface resumes.

If for some reason you need to periodically switch interfaces, then to get around this fprobe problem, you can add lines to the /etc/network/interfaces file after the interface description

Up service fprobe start down service fprobe stop

For example, it might look something like:

Allow-hotplug eth0 iface eth0 inet static address 192.168.0.1 netmask 255.255.255.0 # fprobe up service fprobe start down service fprobe stop

Now, when using utilities that re-read this file (ifup and ifdown), fprobe will exit correctly and start automatically when the interface state is switched. In this case, when the system starts, fprobe will try to start twice (once in the standard way, the other through /etc/network/interfaces), and although it will not succeed, it is still better to remove the autostart of the fprobe system daemon for the sake of order:

sudo update-rc.d -f fprobe remove

Multiple fprobe daemons on one computer

If you want to run several fprobe daemons on one computer at once (to listen to several interfaces), then you will have to slightly change the startup scripts.

First, edit the existing startup script /etc/init.d/fprobe , replacing the line in it

NAME =fprobe

per line

NAME="fprobe"

And besides this, the line

DAEMON_OPTS="-i

per line

DAEMON_OPTS = "-l 1:0 -i $INTERFACE $OTHER_ARGS $FLOW_COLLECTOR "

The first number of the added startup option -l ("1") tells fprobe to use syslog for logging, and the second is the process ID. It is the second number that needs to be changed when running several copies of fprobe , and it should also be in square brackets in the NAME parameter of the corresponding starting script.

Now, to create all the necessary parameters and scripts to run a second copy of fprobe , copy /etc/init.d/fprobe to /etc/init.d/fprobe_1:

sudo cp /etc/init.d/fprobe /etc/init.d/fprobe_1

And /etc/default/fprobe to /etc/default/fprobe_1:

sudo cp /etc/default/fprobe /etc/default/fprobe_1

Edit /etc/default/fprobe_1 to work with the second interface.

Now we need to edit the start script /etc/init.d/fprobe_1 for the second fprobe process. Replace the numbers in the NAME and DAEMON_OPTS parameters with the ones you need:

NAME ="fprobe" DAEMON_OPTS ="-l 1:1 -i $INTERFACE $OTHER_ARGS $FLOW_COLLECTOR "

Additionally, change the configuration file loading code to refer to the fprobe_1. Instead of something like this:

if [ -f / etc/ default/ fprobe ] ; then. / etc/ default/ fprobe fi

It should look something like:

# Include fprobe defaults if available if [ -f / etc/ default/ fprobe_1 ] ; then. / etc/ default/ fprobe_1 fi

Also, newer versions of Ubuntu contain a special comment for the boot system at the beginning of this file. It starts with the line

# ## BEGIN INIT INFO

In this comment you also need to replace the line

# Provides: fprobe

per line

# Provides: fprobe_1

Make sure everything works as it should. To do this, run both fprobes, then look into any process manager (for example, htop) and find both daemons there.

If everything is fine, then add the second fprobe daemon to startup with the command

sudo update-rc.d fprobe_1 defaults

Of course, if you specify the launch of the second fprobe daemon in /etc/network/interfaces , then you should not add it to startup.

This way you can start as many fprobe daemons as you want, just by changing the identifier.

Nfdump Collector and Nfsen Data Visualizer

Nfdump And Nfsen is a collector with accompanying utilities and a web interface for visualizing the collected data. Nfdump is available in the standard Ubuntu repositories, and Nfsen can be downloaded from the official website. You can also see screenshots there to get an idea of ​​what Nfsen is.

You can use any available sensor for this combination, for example fprobe or softflowd.

Nfdump can be installed through any package manager, for example, with the command

sudo apt-get install nfdump

Nfsen needs to be unpacked somewhere on a server from a source archive and a few simple operations need to be performed. First you need to install all the packages needed for work:

sudo apt-get install apache2 libapache2-mod-php5 librrds-perl libmailtools-perl

After that, go to the etc/ folder in the directory where you unpacked Nfsen, and rename the nfsen-dist.conf file located there to nfsen.conf. This is the main Nfsen configuration file. It needs to be edited to suit your needs. Below are just the options you need to change for a basic Nfsen configuration:

# Directory where Nfsen will be installed.$BASEDIR = "/srv/nfsen" ; # If we want to put the web interface in the same folder as everything else$HTMLDIR = "$(BASEDIR)/www" ; # Location of Nfdump utilities for Ubuntu and Debian$PREFIX = "/usr/bin" ; # User to run nfcapd - collector daemon$USER = "www-data" ; # User and group to run the web interface$WWWUSER = "www-data" ; $WWWGROUP = "www-data" ; # Data collection sources - name, local port to listen on, graph color%sources = ( "router1" => ( "port" => "9995" , "col" => "#0000ff" , "type" => "netflow" ) , "router2" => ( "port" => "9996" , "col" => "#00ff00" , "type" => "netflow" ) , ) ;

Detailed documentation on available features can be read at.

Ubuntu 12.04 LTS with Perl 5.14 and nfsen 1.3.6p1 does not have import Socket6. In the files "libexec/AbuseWhois.pm" and "libexec/Lookup.pm" replace:

use Socket6;

Socket6-> import (qw (pack_sockaddr_in6 unpack_sockaddr_in6 inet_pton getaddrinfo) ) ;

After editing the config, you can install Nfsen. To do this, just from the directory where you unpacked it, run the command

./ install.pl etc/ nfsen.conf

All that's left is to start Nfsen. In the example above, the path for installing Nfsen is /srv/nfsen/. In this case, you can use the command to run

/srv/nfsen/bin/nfsen start

To integrate Nfsen into the system as an automatically launched service, run two commands like this:

ln -s / srv/ nfsen/ bin/ nfsen / etc/ init.d/ nfsen update-rc.d nfsen defaults 20

Now, if you left the web server settings at default, the web interface should be available at http://yourserver/nfsen/nfsen.php.

Please note that information will not begin to be collected immediately, so it will take some time before anything appears on the charts. And of course, the network must have sensors that send information to the specified ports.

To monitor connection speed and traffic, you can also use the standard cairo-dock applet - netspeed and the popular system monitoring tool conky.