| |
| |
| |
| Preview |
|
Screen shots |
| |
| |
|
Below are six screen shots. They were all taken during a typical session running edBlackList, using sample Postfix log, Access and Recipients files. |
| |
| |
|
The first screenshot reveals what the user will see when the program is first launched. The user is presented with a traditional GUI window that contains a menu bar, dockable toolbar, a status bar, and a number of collapsible color-coded viewing frames, by which the user will interact with the program. All but one frame have a collapsing button located on their left side (similar to the Mozilla toolbar hiding buttons). This allows the user to reallocate viewing real estate as needed while progressing through the analysis tasks. Only the essential frames are open when the program is first started, and additional frames will open as the tasks are performed. |
| |
| |
|
 |
| |
|
View of edBlackList upon successful launch |
| |
| |
|
The first task is to open the SMTP (e-mail) log file so edBlackList can read its contents. The second screenshot shows how the ListView is revealed and populated with the sorted contents of the log file. The filters frame is opened once a log file is read, since the filter allows one to locate rows based on criteria, such as IP address or DNS address. Also, the statistics frame has been updated with counts after processing the SMTP log file. |
| |
| |
|
 |
| |
|
The ListView widget reveals the contents of a SMTP logfile sorted by processor thread |
| |
| |
|
The second task is to open the blacklist file (specifically named 'access' for the Postfix SMTP server) so edBlackList can read its contents. The third screenshot notes this only by filling in the physical filename of the blacklist file and updating the counts in the statistics frame. One will note that the data being displayed to the used is extensive. Some SMTP files can be many tens of thousands of lines long if not longer. edBlackList was written to assist in analyzing these large amounts of data in order to extract new rules for the blacklist. This is why the frames were designed to be collapsible. Note that the screenshots are for windows that have not yet been maximized, so more viewing area may be available, especially on workstations with large monitors. |
| |
| |
|
Optionally, a DNS destination blacklist may be opened (specifically named 'recipients' for the Postfix SMTP ). This allows edblacklist to segregate blacklist entries by source and destination. This means that the blacklist file read in previously is used for source addresses. |
| |
| |
|
 |
| |
|
edBlackList after having opened all of the prerequisite files for analysis |
| |
| |
|
At this time one can review the SMTP log file contents by perusing the data in the ListView widget. The fourth screenshot shows the user in this instance closed the file selection frame to allow for the ListView widget to be enlarged. Since there is a large amount of information collected for each message, the ListView has to list many columns. To assist the user in viewing the analyzed data, a mouseover window is provided, which summarizes the results of a single row into a pop-up box. The below screenshot shows such a mouseover in use. |
| |
| |
|
 |
| |
|
edBlackList analyzing the log files and showing a mouseover in action |
| |
| |
|
The first step in having edBlackList process the log and blacklist files is to have it analyze the log file. It will determine how each message was processed and see whether the addresses already collected appear in the blacklist. Pressing the Build button on the toolbar will initiate the build task. The fifth screenshot below presents the results of such a build. The most obvious result one will see is that certain checkboxes are marked, representing those addresses that already appear in the blacklist. Note also that the Working frame has now opened up. |
| |
| |
|
 |
| |
|
edBlackList after the Build button was pressed |
| |
| |
|
The working frame allows the user to collect new entries for the blacklist, based upon the desired criteria. For instance, one can collect all of the C subnet IP addresses of messages that were rejected for other reasons (typically the source DNS address), but only those C subnets that did not yet appear in the blacklist file. |
| |
| |
|
Once criteria are selected, the user would press the Execute button located on the toolbar. At this time the edBlackList program would look for new entries that could be used for the blacklist. As the sixth screenshot below shows, the Output Frame would open up, reveling a TextView widget. The results of the criteria processing are listed in this TextView widget, allowing the user to review and edit the results. Usually one would look for false positives at this time. The proposed entries would be ready for inclusion in a blacklist file, (at least for a Postfix SMTP server). |
| |
| |
|
 |
| |
|
edBlackList after the Execute button was pressed |
| |
| |
| |
| |
| Background |
|
History |
| |
| |
|
edBlackList was written to automate the tedious task of analyzing eMail log files for false positives, where messages were bounced erroneously because the message matched an existing rule for a previous spam incident. This analysis task was consuming more and more time each day, to the point that it became untenuous to perform manually. Along the same lines, searching for new spamming incidents to add to the blacklist was beginning to take its toll on the working day, reducing our productivity. |
| |
| |
|
Since Postfix runs on Linux, it was decided that the analysis tool should run on that platform too. Now that the second version of GTK has run through a couple of revisions, it was deemed stable enough to use it to write the edBlackList analyzer. The premier version of edBlackList is being written on Fedora Core 1 distribution, but there is no reason why it could not be compiled to run on other Linux distributions, as long as it has the version 2.4 GTK / Gnome libraries available on the system. |
| |
| |
|
Note that edBlackList was written to run completely separate from Postfix. It is not necessary to run it on the same server as Postfix. It also has no requirements for Postfix code or libraries. All that is required are the log files and the Postfix Access Table file. This was done purposefully, so that the eMail server is not bogged down with additional load of checking messages other than what it was designed for. |
| |
| |
|
Using a Blacklist to filter e-Mail messages |
| |
| |
|
Running one's own eMail server nowadays is a challenge to the small business owner. The greatest burden is dealing with the overwhelming amount of traffic directed to the email server that is either not meant to terminate at the server (relays), or contains unsolicited messages (spam). The undesired traffic consumes bandwidth and resources that has no relation to the business one is trying to run. The unsolicited messages consume server computing and storage resources not budgeted for. Day in and day out we suffer over 65% of the SMTP traffic as having no business finding its way into our server (your mileage may vary). That does not include the messages that are actually accepted and routed to our end users, who then have to process messages for validity or not. |
| |
| |
|
So how do we know at the server, without end user involvement, that 65%+ traffic has no business finding its way to our Postfix server? By periodically reviewing the Postfix logs, one can insert the addresses of misbehaving transmitters into the Postfix Access Table with a REJECT attribute. Going forward, Postfix will not even process messages from miscreants with a previous history of spamming. While Postfix also can be set to ignore relay requests, it also records these entries in the logs, making their addresses available to add as potential spam providers that need to be blocked. |
| |
| |
|
Forcing a reject at the sever provides two advantages for us: |
| |
| |
|
1- Stops the undesired message at the SMTP command level. The spammer does not get a chance to transmit the message to our server, freeing up bandwidth. Note that on occasion some spam SMTP agents do not understand the REJECT command and will transmit some of the header data anyway. Postfix will note the attempt in the logs, but will ignore the attempt at continuing the undesired transmission. |
| |
| |
|
2- Informs the transmitting server that they are not welcome to transmit messages to our server. Addresses appearing in the Postfix Access Table with a REJECT disposition will inform Postfix to transmit a 554 NAK to the originating SMTP server, notifying it that the attempt to transmit to our server failed because the transmitting address has been blocked. The transmitting server should interpret the code to mean go away, we do not want you here, ever! Legitimate SMTP servers will properly recognize the request, while spamming servers do not and will retry many times over, even trying alternate internet routes. |
| |
| |
|
Blocking messages by the end user with filters and junk mail lists is not the same task as having the server block messages based upon address information. Rejecting messages based upon addresses at the server level is a low resource method of blocking spam, but only works if historical records are kept. In other words, one spam message from an address must be received in its entirety in order to categorize it as spam, so that the address can then be added to the Postfix Access Table. Depending on how lose the rules are applied, one could add large subnets of IP addresses based on a single spam incident, and block large numbers of potential spam. Problem is that false positives can result, where a legitimate address may reside within a block of spammers. So the logs must be reviewed constantly for the false positives, as well as new candidate addresses to add to the Access Table. |
| |
| |
|
The end user approach to filtering spam with junk mail lists and the like, run by SMTP clients (such as Outlook), are resource intensive, and best handled on computers other than the eMail server. These programs typically look inside the message payload and inspect it against various rules and patterns. These filtering methods should not be discarded if server level blocking is deployed, rather used in conjunction with it. Note that the 65%+ figure presented above does not represent all spam directed towards our eMail server. That is the measurable amount of traffic we are blocking based upon historical information. Our end users are then handling spam that "slips" past our filters. We do not have accurate metrics on those numbers, but ad hoc reports that half the end user messages are spam is not unheard of. |
| |
| |
|
Understanding SMTP message blocking |
| |
| |
|
In order to successfully block spam, one must know what addresses they are using for the connections. Unfortunately, there are large blocks of addresses that will need to be blocked. True spammers do not issue messages from well-known addresses, as these activities are illegal, and they would be shut down quickly. Rather, spam is transmitted by nefarious people who have built up robot networks of computers, created through the use of viruses and trojans, along with taking advantage of defects shipped by the largest volume OS provider. Computers left turned on and having fulltime internet connectivity are most vulnerable. The spammers will then use groups of these computers as zombies, sending messages in campaigns. The groups of computers are rotated, so the same ones are not continuously used, else they would raise suspicion, possibly being detected and remediated. |
| |
| |
|
We are currently blocking 5068 B level subnets, which works out to be 332,136,448 computer addresses. We also block over 36,000 named DNS domains. The majority of these addresses are the public DHCP addresses allocated to the major ISP providers (such as AOL, ATT, Comcast, Roadrunner, Southwest Bell, etc). The hardest part is obtaining the legitimate SMTP server addresses that SMTP mail is expected to be issued from. For the most part (but not always), the ISPs police their SMTP server traffic and prevent spam from being transmitted by their legitimate servers. |
| |
| |
|
So blocking spam from a robot network requires the SMTP administrator to analyze the logs for traffic from computers that seem to be a home user's PC, which should be emanating instead from a recognized SMTP address. The real SMTP servers will have proper PTR records registered in name servers for the IP address used, with names like "MX.ISP.NET", "OUTBOUND-SMTP.ISP.COM", or the like. Addresses such as "reversed.ip.subnets.ISP.NET" are resulting from PCs running in a DHCP pool. Note that one source of false positives are small businesses that run their own SMTP server with unregistered domains (all to save a few bucks). Nowadays, more and more SMTP servers are setup to reject messages from domains without a PTR record, so maybe that will come to an end through harassment of rejected messages. |
| |
| |
|
One can also block on the DNS name that may be used in the SMTP "From" field. Since this field is completely at the control of the eMail message author, this field may not be much of much value to block. |
| |
| |
| |
| Release Schedule |
|
Trial Version 0.1 edBlackList Release [soon-to-be latest release] |
| |
| |
|
This version is now being written. Stay tuned for the posting of this code, will happen shortly. |
| |
|
It is intended that this program will be posted to SourceForge, assuming that they will have us there. |
| |
| |
| |
| Downloads |
|
Prerequisites: |
| |
| |
|
1- GTKMM (we used 2.4) |
| |
|
>> last accessible from here |
| |
| |
|
2- GLADEMM (we used version 2.4) |
| |
|
>> last accessible from here |
| |
| |
|
Latest files |
| |
|
Coming soon! |
Legal