FailLog [1.1.0.0]

[ImportBot]

Originally Posted by MorpheusX(AUT)*:

 

Fail Log

 

 

For BF3, this plugin logs game server crashes, layer disconnects and Blaze dumps.

The plugin was (beside a small webrequest-part) developed by PapaCharlie9, so credits to him

We've added a central database at our development server which collects webreports sent in by the plugin. In the future, we'll also provide a webbased form to submit "manual" BlazeReports and (most likely) a "view last BlazeReports" page.

 

 

Description

---

Each failure event generates a single log line. The log line is written to plugin.log. Optionally, it may also be written to a file in procon/Logs and/or to a web logging database on myrcon.com, controled by plugin settings (see below). Note that this plugin must be run without restrictions (not in sandbox mode) in order to use either the optional separate log file or web log features. The plugin may be run in sandbox mode if both of the optional logging features are disabled.

 

 

The contents of a log line are divided into fields. The following table describes each of the fields and shows an example:

Type

A label that describes the type of failure. The types tracked are game server restarts, Procon disconnects, blaze disconnects, and network congestion to/from Procon.

 

 

UTC

UTC time stamp of the plugin's detection of the failure; the actual event might have happened earlier

 

Server

Server name per vars.serverName

 

Map

Friendly map name

 

Mode

Friendly mode name

 

Round

Current round/total rounds

 

Players

vars.maxPlayers/previous known player count/current player count

 

Uptime

Uptime of game server as days.hh:mm:ss

 

Details

All of the information you entered in Section 2 of the settings, plus the Region and Country from serverInfo

 

 

 

 

Blaze Disconnect Failures

 

This plugin uses a heuristic (a guess) to decide if a loss of players indicates a Blaze disconnect failure. The loss of players is calculated on every admin.listPlayers event. These events happen at least once every 30 seconds, but may happen more frequently if you run other plugins. This means that detection of Blaze events is very dependent on your configuration. You may need to adjust the settings of this plugin to detect Blaze disconnects accurately.

 

 

Settings

---

Plugin settings are described in this section.

 

 

Section 1

General plugin settings.

 

Debug Level: Number from 0 to 9, default 2. Sets the amount of debug messages sent to plugin.log. Caught exceptions are logged at 3 or higher. Raw event handling is logged at 8 or higher.

 

Enable Log To File: True or False, default False. If False, logging is only to plugin.log. If True, logging is also written to the file specified in Log File.

 

Log File: Name of the file to use for logging. Defaults to "fail.log" and is stored in procon/Logs.

 

Enable Web Log: True or False, default True. If False, no logging is sent to the web database. If True, logging is also sent to the web database maintained by Procon staff.

This feature is deprecated, so set to false.

 

Blaze Disconnect Heuristic Percent: Number from 33 to 100, default 75. Not every sudden drop in players is a Blaze disconnect. Also, sometimes a Blaze disconnect does not disconnect all players or they reconnect before the next listPlayers event happens. This heuristic (guess) percentage accounts for those facts. The percentage is based on the ratio of the count of lost players to the last known count of players. For example, if you set this value to 75, it means any loss of 75% or more players should be treated as a Blaze disconnect. If there were 32 players before and now there are 10 players, (32-10)/32 = 69%, which is not greater than or equal to 75%, so no Blaze failure. If there were 32 players before and now there are no players, (32-0)/32 = 100%, a Blaze failure. If you want to only detect drops to zero players, set this value to 100. If the last known player count was less than 12, no detection is logged, even though a Blaze disconnect may have happened. See also Blaze Disconnect Window Seconds.

 

Blaze Disconnect Window Seconds: Number from 30 to 90, default 30. Normally, listPlayers events happen every 30 seconds and that is normally enough time to detect a Blaze disconnect. However, if you have lots of other plugins running, listPlayer events may happen more frequently than every 30 seconds, which may not be enough time to detect a large enough loss of players. Even if the interval between events is 30 seconds, sometimes a Blaze disconnect takes longer than 30 seconds to complete. This setting allows you to adjust the plugin to handle those situations. If you notice loss of players that you suspect are Blaze disconnects but no failure is registered, increase this value. Try 60 at first and if that isn't enough, add 15 seconds and try again, until you get to the max of 90 seconds.

 

Enable Restart On Blaze: True or False, default False. If True, the game server will be restarted with an admin.shutDown command when a Blaze disconnect is detected and the remaining number of players is zero. Use with caution!

 

Restart On Blaze Delay: Number, default 0. Time in seconds to wait before invoking the admin.shutDown command after a Blaze disconnect. Use with caution, since most servers get messed up or don't save progress properly after a Blaze disconnect, so instant restarts would be advised. Setting it to 0 instantly executes the command.

 

Enable Email On Blaze: True or False, default False. If True, the plugin will send a notification-email if you server blazes (see settings below). Make sure to disable the sandbox or allow SMTP-connections and your mailserver + mailserver-port in the trusted hosts.

 

 

Section 2

These settings fully describe your server for logging purposes. Information important for tracking global outages and that can't be extracted from known data is included. All of this information is optional.

 

Game Server Type: Type of game server, defaults to BF3.

 

Ranked Server Provider: Name of the RSP/GSP that hosts your game server.

 

Server Owner Or Community: Your name or the name of your clan, whoever runs the server. Technically you are a 'renter' rather than an owner, but you are responsible for the game server.

 

Contact Info: Email address, clan website or your myrcon.com forum user name, to follow-up with you if additional information is needed.

 

Server Region: The Country and Region known by serverInfo in automatically included, but that information is very high level, su as NAm/US. Use this setting to specify the city, state or to narrow down the region further.

 

Additional Information: These are additional details that were not anticpated at the time of writing of this plugin but that may prove useful in the future.

 

 

Section 3

These settings configure the BlazeReport-mail being sent. The following values can be entered as wildcards at the email-subject and email-body and will be replaced: %servername%, %serverip%, %serverport%, %utc%, %players%, %map%, %gamemode%, %round%, %uptime%.

 

 

Email Recipients: List of email-addresses to send the notifications to, one each line.

 

 

Email Sender: Email-Address being displayed in the 'From:' field.

 

 

Email Subject: Subject of the notification-email. You can use the values listed above to add information about the BlazeReport.

 

 

Email Message: Body of the BlazeReport-email, can be fully styled with HTML. You can use the values listed above to add information about the BlazeReport.

 

 

SMTP Hostname: Hostname/IP-Address of the SMTP-server used to send email.

 

 

SMTP Port: Number between 0 and 65535, default 25. Port of the SMTP-Server used to send email.

 

 

SMTP Use SSL: True of False, default false. Toggles the usage of SSL for the connection to your SMTP-server.

 

 

SMTP Username: Username used to identify with your SMTP-server.

 

 

SMTP Password: Password used to identify with your SMTP-server.

 

 

Development

---

This plugin is an open source project hosted on GitHub.com. The repo is located at

https://github.com/PapaCharlie9/fail-log and the master branch is used for public distributions. See the Tags tab for the latest ZIP distributions. If you would like to offer bug fixes or new features, feel free to fork the repo and submit pull requests.

 

 

Download

---

https://github.com/PapaCharlie9/fail...ve/1.1.0.0.zip

Edited August 7, 2019 by maxdralle

[ImportBot]

Originally Posted by Sabn0ck*:

 

Cool but what's the purpose other than a server restart after Blaze dc?

[ImportBot]

Originally Posted by PapaCharlie9*:

 

Cool but what's the purpose other than a server restart after Blaze dc?

It's to help diagnose problems with your server. If you are getting unusual crashes, disconnects, network lag or blaze disconnects, this plugin logs the time and details of the failure. You can use this log to file problem reports with your GSP.

 

The web logger allows Myrcon community members to share their failure reports, in order for DICE or EA to collect information to diagnose system-wide failures. At the very least, you will be able to find out if your problems are just you or if everyone in the world is having them.

 

Inspired by the request for help and discussion in this thread: showthread....s-Ideas-please*!

[ImportBot]

Originally Posted by Sabn0ck*:

 

So the info logged from this plugin will be accepted by DICE?

[ImportBot]

Originally Posted by PapaCharlie9*:

 

So the info logged from this plugin will be accepted by DICE?

I'd put it a different way. If a bunch of people think it's a system wide problem and someone asks DICE to investigate, the first thing DICE is going to ask for is some data for why we think it is a system wide problem. The web log would provide that data without anyone having to do anything other than install the plugin and enable it.

[ImportBot]

Originally Posted by Sabn0ck*:

 

That could be true but I believe the first thing they will do is ask, "what RSP did this come from."

That leads me to wondering if RSP's will even accept the data logs(they'll accept for sure) and pass them along to DICE(this is what won't happen).

I'm unsure of the need to run this but I'll give it a shot. Thanks for the work

[ImportBot]

Originally Posted by MorpheusX(AUT)*:

 

That could be true but I believe the first thing they will do is ask, "what RSP did this come from."

That leads me to wondering if RSP's will even accept the data logs(they'll accept for sure) and pass them along to DICE(this is what won't happen).

I'm unsure of the need to run this but I'll give it a shot. Thanks for the work

There is no real "need" to run this unless you simply want to share information about crashes with fellow admins, like suggested in the thread PC9 linked. If/how DICE or a GSP can make use of this will be seen in the future, it doesn't harm to gather some information for them and maybe also detect a certain "pattern" or similar.

[ImportBot]

Originally Posted by Ruger*:

 

Thanks MorpheusX(AUT). Got it running on both mine now.

[ImportBot]

Originally Posted by a928396*:

 

Thank you for this plugin - just installed

[ImportBot]

Originally Posted by xFaNtASyGiRLx*:

 

Enable Restart On Blaze: True or False, default False. If True, the game server will be restarted with an admin.shutDown command when a Blaze disconnect is detected and the remaining number of players is zero. Use with caution!

Is this the same as the Automatic Round Restarter Plugin? Will it restart the server if there was no blaze but everyone just left and the number of players reached 0?

 

Do we get to view this database of collected blazes or maybe a graph?

 

This would be great if all servers ran this plugin.

[ImportBot]

Originally Posted by EBassie*:

 

Is this the same as the Automatic Round Restarter Plugin? Will it restart the server if there was no blaze but everyone just left and the number of players reached 0?

No, only when the plugin detected a blaze, the option "Enable Restart on Blaze" set to true and the server has 0 players.

 

Do we get to view this database of collected blazes or maybe a graph?

Probably, yes. I'm currently working on a manual submission page for admins who don't want to use this plugin. (but why not use as it saves much time and you get a neat auto-restarter )

 

A next step could be a page with overview of latest blazes, or blazes per region, per GSP etc.

 

This would be great if all servers ran this plugin.

Yes, indeed. The more info gathered the better.

[ImportBot]

Originally Posted by Flaxe*:

 

Enabled it now on our 2 public servers, both located in Frankfurt, Germany.

I'm unsure if it will work good to enable this on a match server... since it will usually be maximum 10 players on it and everyone will leave at the same time (when match is finished) it will probably look like a Blaze according to the plugin?

 

Great job, thanks for the plugin and your efforts!

[ImportBot]

Originally Posted by EBassie*:

 

I'm unsure if it will work good to enable this on a match server... since it will usually be maximum 10 players on it and everyone will leave at the same time (when match is finished) it will probably look like a Blaze according to the plugin?

If I'm reading this correct, your 10 player match should have no issue when all players disconnect at the same time:

 

If the last known player count was less than 12, no detection is logged, even though a Blaze disconnect may have happened.

[ImportBot]

Originally Posted by PapaCharlie9*:

 

I'm unsure if it will work good to enable this on a match server... since it will usually be maximum 10 players on it and everyone will leave at the same time (when match is finished) it will probably look like a Blaze according to the plugin?

EBassie is correct and it is also a good point. The plugin is not recommended for unranked/match servers, even if they do go above the minimum 12 players.

[ImportBot]

Originally Posted by tarreltje*:

 

Thnx for the plugin, I have it running now!! Also nice feature to restart te server after a blaze. Yesterday i had 2 times a blaze. Server Came back out of it self but people could use only 1 weapon each kit.... Only a restart would work!

[ImportBot]

Originally Posted by Phogue*:

 

May I suggest sending along a snapshot of the current server settings? As an example, it would be interesting to see that only servers with friendly fire enabled are having problems and what not.

 

Bonus points if you had a snapshot of the current settings along with any other settings that have been modified in the last 10 seconds or so. It could be a server setting being changed that is causing problems for instance.

 

Double bonus points if you send through a before/after player list as well.

 

If it's not already then perhaps the additional information can be sent through as xml/json so you can expand/shrink the data being passed through in the future without breaking a comma-separated index.

[ImportBot]

Originally Posted by MorpheusX(AUT)*:

 

May I suggest sending along a snapshot of the current server settings? As an example, it would be interesting to see that only servers with friendly fire enabled are having problems and what not.

 

Bonus points if you had a snapshot of the current settings along with any other settings that have been modified in the last 10 seconds or so. It could be a server setting being changed that is causing problems for instance.

 

Double bonus points if you send through a before/after player list as well.

 

If it's not already then perhaps the additional information can be sent through as xml/json so you can expand/shrink the data being passed through in the future without breaking a comma-separated index.

My ideas where to send server-settings, player list, procon-plugins(as far as possible) and the latest X lines of chat+console... thinking about a proper way to transmit that, maybe as a huge base64 encoded string to save some space.

[ImportBot]

Originally Posted by IAF-SDS*:

 

For convenience and to keep it in the relevant thread, I brought this over from the "Blaze/EA/Server Drops" thread (I originally posted there before I read this topic):

 

Morpheus,

 

Is there a way for you to add your NotifyMe capability within this Blaze Disconnects Detector plugin so that it can automatically email a predefined message to the admins when it detects a blaze disconnect?

 

Often, when the server blazes, it will either require a restart to get perks working again or admins to help seed it again.

 

It would be great if this plugin could email us. What do you think?

I thought about emailing at Blaze, might add in the future

Regarding the restart: you can have the plugin do that automatically, check the settings/description

This would be a great feature to email admins on blaze disconnects.

 

I see it only restarts if the number of players are zero, which is a good configuration since sometimes blaze dumps only 2/3 of the players.

 

Good work Morpheus and Papa.

[ImportBot]

Originally Posted by MorpheusX(AUT)*:

 

For convenience and to keep it in the relevant thread, I brought this over from the "Blaze/EA/Server Drops" thread (I originally posted there before I read this topic):

 

This would be a great feature to email admins on blaze disconnects.

 

I see it only restarts if the number of players are zero, which is a good configuration since sometimes blaze dumps only 2/3 of the players.

 

Good work Morpheus and Papa.

Thanks for posting this over here as a reminder, will look into this as the first patch next week. Shouldn't be too hard since the NotifyMe code should be usable quite much as it is.

[ImportBot]

Originally Posted by Flaxe*:

 

If I'm reading this correct, your 10 player match should have no issue when all players disconnect at the same time:

That's very correct Now when you say it I remember that I read that in the first post, my fault

 

EBassie is correct and it is also a good point. The plugin is not recommended for unranked/match servers, even if they do go above the minimum 12 players.

Ok, I see.

Thanks both of you.

[ImportBot]

Originally Posted by xFaNtASyGiRLx*:

 

Having a discussion with a couple of other admins and some good points were raised.

 

First, I think for this to work well is if MANY admins from ALL GSPs used this plugin.

 

I think the threshold should be SET to 100 and nothing lower. Because if a bunch of players choose to leave at once due to a baserape- that is not a blaze but the plugin will count it as a blaze.

 

I also think that having the ability to see the database or statistics would encourage more admins to use this.

[ImportBot]

Originally Posted by Hutchew*:

 

Sweet!

 

Great step towards fixing the problems we suffer from.

 

Hutchew

[ImportBot]

Originally Posted by tarman*:

 

What debug level do you recommend for diagnosing the current situation?

[ImportBot]

Originally Posted by MorpheusX(AUT)*:

 

What debug level do you recommend for diagnosing the current situation?

DebugLevel 3 shows reports being sent, but that's not reall necessary since you usually see the server blaze at the same time

[ImportBot]

Originally Posted by PapaCharlie9*:

 

Double bonus points if you send through a before/after player list as well.

Before/after a failure, or before/after settings change?

 

Plugins or changing settings on plugins are probably more likely to create some kind of game server problem, but that's a lot of work. Is every plugin's GetPluginVariables registered and visible to every other plugin? Can they be enumerated with this.GetRegisteredCommands()?

[ImportBot]

Originally Posted by tarman*:

 

DebugLevel 3 shows reports being sent, but that's not reall necessary since you usually see the server blaze at the same time

Just sent my first blaze report:

Code:

[20130511_20:24:47] 93.93.65.249_47400: Type:BLAZE_DISCONNECT, UTC:20130511_18:24:47, Server:"EL MANICOMIO 1 CLAN (BHR) CONQUEST[64] :: Iberbit.com ", Map:Operation Firestorm, Mode:Conquest Large, Round:1/1, Players:0/54/0, Uptime:3.00:15:11, Details:"BF3,iberbit.com,[BHR],www.clanbhr.com,Spain,EU/ES,"

I run procon layer with multiple servers, but the log file is located at logs/ folder instead of individual server folder, is this correct?

[ImportBot]

Originally Posted by Phogue*:

 

Before/after a failure, or before/after settings change?

 

Plugins or changing settings on plugins are probably more likely to create some kind of game server problem, but that's a lot of work. Is every plugin's GetPluginVariables registered and visible to every other plugin? Can they be enumerated with this.GetRegisteredCommands()?

Before/after a failure.

 

Just the game server settings (team damage enabled, player limits etc). Just stuff about the game server. Ignore Procon.

[ImportBot]

Originally Posted by Jaythegreat1*:

 

Got this running to, think we already sent 1 report to.

[ImportBot]

Originally Posted by Jaythegreat1*:

 

One suggestion... instead of instantly restarting when detecting a "Blaze" can it wait x amount of seconds and see if players re-join the server. While it's nice to fix the issue of players not able to re-join, if players can re-join after "Blaze" the server tends to re-fill faster. If the server automatically restarts, it tends to kill the server.

 

So what I'm getting at, is if it detect players are re-joining, it still sends a report but cancels restart.

[ImportBot]

Originally Posted by dyn*:

 

I will have this downloaded, installed, and ready to share data with the world. This will truly be great to finally know exactly when other people are crashing and what their specs are (host, location, size, etc).

 

I don't know about other GSPs, but with ours (NFO) we know the name of the server we're hosted on. If we could also include that tid bit of info in the generated report it could further help.

 

Then 'just' (pretty please? :P ) throw up some type of method for us to be able to view the data and I would love y'all for ever. Even if it doesn't help DICE that much (though it should), we admins will now know if it really is "just us" or there are other people crashing too.