Jump to content

[BF4] Stats webpage for XpKiller's Stats Logger Plugin


tyger07

Recommended Posts

Originally Posted by ty_ger07*:

 

I am not sure if you guys have noticed, but you can do fuzzy date time searches on the chat log search page such as '12:00 yesterday' or 'last month' or '2 weeks ago'. I hope you guys enjoy that feature. If you enter a time, it will search for 10 minutes within that time. If you enter a day with no time, it will show all day. If you enter a week with no time, it will show all week. And so on.

* Restored post. It could be that the author is no longer active.
Link to comment
  • Replies 1.7k
  • Created
  • Last Reply

Originally Posted by ty_ger07*:

 

It's quite clear that XpKiller's session stats are very different than each player's server stats. I don't know why they are so different. I know this issue has been complained about before, but I guess XpKiller has not responded. :S

I guess if all players' session stats are equally screwed up, they will at least still provide a comparison.

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

It has come to my attention that I have put in nearly 1700 hours of overtime so far this year. Essentially, that means that I am working the equivalent of two full time jobs at once. This is my excuse for my slow updates.

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ColColonCleaner*:

 

It has come to my attention that I have put in nearly 1700 hours of overtime so far this year. Essentially, that means that I am working the equivalent of two full time jobs at once. This is my excuse for my slow updates.

Damn son, putting in work! Hope the extra hours are paying off though :smile:
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

Thanks for the update

Wow, your database has gotten huge. The combined chat page, for example, takes 20+ seconds to load because it has to count 1 million+ rows every time.

 

Have you considered pruning it down a bit? I have been messing around a bit with a stats database cleaner/optimizer which I may release for people like you.

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by dyn*:

 

Have you considered pruning it down a bit? I have been messing around a bit with a stats database cleaner/optimizer which I may release for people like you.

tbl_chatlog 2.1 GiB

 

Showing rows 15280950 - 15280973 ( 15,280,974 total, Query took 20.0390 sec)

 

That would be mighty helpful! Looking through the chat I do see old server spam messages in there and then the ID_CHAT (COMM ROSE) messages. I'm sure just that stuff takes up a ton of space.

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Prophet731*:

 

Wow, your database has gotten huge. The combined chat page, for example, takes 20+ seconds to load because it has to count 1 million+ rows every time.

 

Have you considered pruning it down a bit? I have been messing around a bit with a stats database cleaner/optimizer which I may release for people like you.

Is that for pagination? What is the query that took 20 seconds? Ours is 2.78GB with ~25,821,404 rows. There is a missing index for the logDate column and if your searching on that it will take a while. After I added the index to ours, datetime searching was drastically reduced.

 

Run this query on the database and tell me if it speed it up for you. It might take a while on a large table.

 

Code:

ALTER TABLE `tbl_chatlog` ADD INDEX (logDate);
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

Is that for pagination? What is the query that took 20 seconds? Ours is 2.78GB with ~25,821,404 rows. There is a missing index for the logDate column and if your searching on that it will take a while. After I added the index to ours, datetime searching was drastically reduced.

 

Run this query on the database and tell me if it speed it up for you. It might take a while on a large table.

 

Code:

ALTER TABLE `tbl_chatlog` ADD INDEX (logDate);
Yeah, it's for counting the number of rows for pagination. It's not my database; I hope you are directing your suggestion towards those who need it. :smile:
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Prophet731*:

 

Yeah, it's for counting the number of rows for pagination. It's not my database; I hope you are directing your suggestion towards those who need it. :smile:

Yeah its for everyone.
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

Yeah its for everyone.

Thank you for the tip. :smile:

Actually, I was thinking -- after I wrote the reply above -- it might be a very important thing I could include in some optimizer code.

 

Edit: I should open my eyes. Now that I see who you are based on your merits, I should be extra grateful for any guidance you are willing to offer to a self-educated PHP/SQL/HTML/CSS/Javascript hack like me.

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Prophet731*:

 

Thank you for the tip. :smile:

Actually, I was thinking -- after I wrote the reply above -- it might be a very important thing I could include in some optimizer code.

Only reason I found out the index was missing was when I was trying to trim down my response times with my live scoreboard and chat. Took 2 seconds for the chat to be return when filtering by date to a remote DB. After adding that index its now less than 1 second so huge increase.

 

All about those indexes on the right columns that make the difference :smile:

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

tbl_chatlog 2.1 GiB

 

Showing rows 15280950 - 15280973 ( 15,280,974 total, Query took 20.0390 sec)

 

That would be mighty helpful! Looking through the chat I do see old server spam messages in there and then the ID_CHAT (COMM ROSE) messages. I'm sure just that stuff takes up a ton of space.

It's really easy to get rid of the commo rose and server spam. Do you know what phpMyAdmin is? If not: phpMyAdmin is a database administration tool every web server host should provide you a link to somewhere in your account which you can use to log into your database, view and alter your database, and issue queries with.

 

Once logged into phpMyAdmin, issue the following queries:

ID_CHAT.png

Code from screenshot above (removes commo rose messages):

Code:

DELETE
FROM `tbl_chatlog`
WHERE `logMessage` LIKE '%ID_CHAT%'
And to remove Server spam:

Code:

DELETE
FROM `tbl_chatlog`
WHERE `logSoldierName` = 'Server'
And to remove blank messages (probably messages in other languages with Cyrillic characters):

Code:

DELETE
FROM `tbl_chatlog`
WHERE `logMessage` = ''
With a huge database, that will probably remove quite a few lines. You may also consider deleting old chat messages if you don't think you will ever look back that far.

 

To remove chat older than 6 months:

Code:

DELETE
FROM `tbl_chatlog`
WHERE `logDate` < DATE_SUB(CURRENT_DATE(),INTERVAL '182' DAY);
After you are all done, you should optimize the table:

Code:

OPTIMIZE TABLE `tbl_chatlog`
And of course, consider 13lackHawk's advice above.
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Prophet731*:

 

ty_ger you're going to get a kick out of this. Out of the ~25 million chat records we have, ~10,464,057 are done by the Server spam. With the commo rose its ~3,990,484. Our server does love to chat don't ya think. haha.

 

Here are the SQL for those that want to see how many entries there are for both of them.

 

Code:

SELECT     FORMAT(COUNT(ID), 0) AS 'Total'
FROM
    tbl_chatlog
WHERE
    `logMessage` LIKE 'ID_CHAT%';
Code:
SELECT 
    FORMAT(COUNT(ID), 0) AS 'Total'
FROM
    tbl_chatlog
WHERE
    `logSoldierName` = 'Server';
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Jamesonp*:

 

Is that for pagination? What is the query that took 20 seconds? Ours is 2.78GB with ~25,821,404 rows. There is a missing index for the logDate column and if your searching on that it will take a while. After I added the index to ours, datetime searching was drastically reduced.

 

Run this query on the database and tell me if it speed it up for you. It might take a while on a large table.

 

Code:

ALTER TABLE `tbl_chatlog` ADD INDEX (logDate);
Thanks! I'm not really seeing much of a difference on page creation times. Also my ADK Web Admin site fails to load the chatlog for players who have been around for a long time. It stalls and Chrome asks me to kill the page.

 

Any other tips for speeding the database up?

 

@ty_ger07

 

I was going through my slow-query log to try and alleviate some of the slow load times and found the following:

 

# Time: 141027 13:35:06

# User@Host: adkatsro[adkatsro] @ localhost [] Id: 12374

# Schema: adkat_statlog_bf4 Last_errno: 0 Killed: 0

# Query_time: 14.003345 Lock_time: 0.000163 Rows_sent: 3 Rows_examined: 4577048 Rows_affected: 0

# Bytes_sent: 499

SET timestamp=1414442106;

SELECT `logDate`, `logSoldierName`, TRIM(`logMessage`) AS Message, `logSubset`

FROM `tbl_chatlog`

WHERE `ServerID` in (4,5,6,7)

AND `logMessage` != ''

AND `logMessage` NOT LIKE '%ID_CHAT_%'

AND `logSoldierName` != 'Server'

ORDER BY logDate ASC, `logDate` ASC

LIMIT 2216880, 20;

The added 5 seconds to make a 20 second load time is this:

 

# User@Host: adkatsro[adkatsro] @ localhost [] Id: 4831

# Schema: adkat_statlog_bf4 Last_errno: 0 Killed: 0

# Query_time: 5.742400 Lock_time: 0.000150 Rows_sent: 1 Rows_examined: 2360132 Rows_affected: 0

# Bytes_sent: 66

use adkat_statlog_bf4;

SET timestamp=1414441655;

SELECT count(`logDate`) AS count

FROM `tbl_chatlog`

WHERE `ServerID` in (4,5,6,7)

AND `logMessage` != ''

AND `logMessage` NOT LIKE '%ID_CHAT_%'

AND `logSoldierName` != 'Server';

Any suggestions, or do I just have too much data?
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

I am not sure if the extra ANDs in the WHERE clause (I mean: AND `logMessage` != '' AND `logMessage` NOT LIKE '%ID_CHAT_%' AND `logSoldierName` != 'Server') slow it down more or not. I suppose they might. You could try removing those ANDs from both queries in chat.php, chat-search.php, and chat-live.php and see if it speeds up some.

 

You coupd also try ", `logMessage` AS Message" instead of ", TRIM(`logMessage`) AS Message". I don't know if TRIM takes a lot of extra time. TRIM just removes any extra spaces before or after a message. But, it shouldn't make much difference since you are only trimming 20 messages (a small number).

 

Otherwise, I think what would help a lot are the queries previously mentioned in order to slim down your chat log. You could probably reduce your chat log size drastically simply by removing the server spam, commo rose commands, and blank messages. Look back a few posts. Don't forget to optimize the table after you are done making changes.

 

The second query (the 5 second one you mentioned) is what counts the total number of rows. I would have thought that the second query should be much slower (20 seconds for example) than the first query (5 seconds for example) since the first query is only gathering data from 20 rows while the second query is gathering data from 2 million + rows (to count the total number of rows).

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Prophet731*:

 

I am not sure if the extra ANDs in the WHERE clause (I mean: AND `logMessage` != '' AND `logMessage` NOT LIKE '%ID_CHAT_%' AND `logSoldierName` != 'Server') slow it down more or not. I suppose they might. You could try removing those ANDs from both queries in chat.php, chat-search.php, and chat-live.php and see if it speeds up some.

 

You coupd also try ", `logMessage` AS Message" instead of ", TRIM(`logMessage`) AS Message". I don't know if TRIM takes a lot of extra time. TRIM just removes any extra spaces before or after a message. But, it shouldn't make much difference since you are only trimming 20 messages (a small number).

 

Otherwise, I think what would help a lot are the queries previously mentioned in order to slim down your chat log. You could probably reduce your chat log size drastically simply by removing the server spam, commo rose commands, and blank messages. Look back a few posts. Don't forget to optimize the table after you are done making changes.

 

The second query (the 5 second one you mentioned) is what counts the total number of rows. I would have thought that the second query should be much slower (20 seconds for example) than the first query (5 seconds for example) since the first query is only gathering data from 20 rows while the second query is gathering data from 2 million + rows (to count the total number of rows).

The query is fine. It's the logMessage column which causes the slow down.
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Jamesonp*:

 

Thanks for the reply, to alleviate some of the issues with too much chat log history I've just set it to allow the last 30 days of chat with the following line:

 

AND logDate > NOW() - INTERVAL 30 DAY

Example:

 

{

$TotalRows_q = @mysqli_query($BF4stats,"

SELECT count(`logDate`) AS count

FROM `tbl_chatlog`

WHERE `ServerID` in ({$ids})

AND logDate > NOW() - INTERVAL 30 DAY

AND `logMessage` != ''

AND `logMessage` NOT LIKE '%ID_CHAT_%'

AND `logSoldierName` != 'Server'

");

Also have you thought about removing the pagination count to gather how many pages total and just have it display only the first 10 pages, then when you click the 10th page it generates the next 10 pages. Just like how Google does it on search results?
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

Is there any reason why you don't want to delete the commo rose ("ID_CHAT...") commands or server spam or empty messages? If you are filtering it to only fetch the last 30 days, is there a reason why you don't want to delete the old messages?

 

Regarding your question:

If you are viewing the newest chat messages on the last page, how do you know which page you are on? If the last page is page 1 and you go back a few pages, are you then on page -2? If I am selecting a row number to start from (offset in query), I need to know valid offsets -- positive integers between 0 and infinity which actually exist in the database. "LIMIT 2216880, 20"

I could make it so the webpage doesn't display the number; but still I feel like I need to know the number.

 

I guess that I could make every query start from page 1 (even if it caused the results to be given in backwards order), but it seems awkward. It would send people the message "enjoy manually scrolling through hundreds of thousands of pages to find the last result".

 

 

I do see a mistake with my leaderboard row count logic for combined stats though. I could definitely optimize fetching that number.

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Jamesonp*:

 

Is there any reason why you don't want to delete the commo rose ("ID_CHAT...") commands or server spam or empty messages? If you are filtering it to only fetch the last 30 days, is there a reason why you don't want to delete the old messages?

We use the database in conjunction with Adkats Server Admin. Deleting the chat history would be counter productive to allowing us to maintain a record of all things which have occurred on our servers and the chat history for players. We use this tool to aid us in deciding who to ban and to view people's history.

 

If you are viewing the newest chat messages on the last page, how do you know which page you are on? If the last page is page 1 and you go back a few pages, are you then on page -2? If I am selecting a row number to start from (offset in query), I need to know valid offsets -- positive integers between 0 and infinity which actually exist in the database. "LIMIT 2216880, 20"

I could make it so the webpage doesn't display the number; but still I feel like I need to know the number.

Instead of reading the amount of rows each time, would it be possible to store the value in a table and have it update every 30 minutes (or next page view if over 30 minutes)?

 

A good reference on how something like this would work is how Google calculates it's results. For example, if you Google "test" you'll see the following message:

 

"About 247,000 results" and it only shows you the first 10 pages. Once you click page 10, it shows you the next 4 pages possible (14 pages total) and so on.

 

If you need to search for a specific period of time you'd use the search logic you already have in place.

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

We use the database in conjunction with Adkats Server Admin. Deleting the chat history would be counter productive to allowing us to maintain a record of all things which have occurred on our servers and the chat history for players. We use this tool to aid us in deciding who to ban and to view people's history.

What about deleting the commo rose spam, server spam, blank messages, and optimizing the table after doing so? You should see a pretty huge boost just by doing that.
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

Is that for pagination? What is the query that took 20 seconds? Ours is 2.78GB with ~25,821,404 rows. There is a missing index for the logDate column and if your searching on that it will take a while. After I added the index to ours, datetime searching was drastically reduced.

 

Run this query on the database and tell me if it speed it up for you. It might take a while on a large table.

 

Code:

ALTER TABLE `tbl_chatlog` ADD INDEX (logDate);
Could you please explain this?

logdate_index.png

 

It appears that I now have two equal indexes for logDate. Doing so actually seems to have slowed down the query a bit. Perhaps its because the borrowed database I am using is from a user who already ran your webpage code or something? Or maybe XpKiller made a change to his plugin at some time?

* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Prophet731*:

 

Could you please explain this?

logdate_index.png

 

It appears that I now have two equal indexes for logDate. Doing so actually seems to have slowed down the query a bit. Perhaps its because the borrowed database I am using is from a user who already ran your webpage code or something? Or maybe XpKiller made a change to his plugin at some time?

That could be. When we ran XpKillers logger for BF4 it re-created the tables for us and so I guess at the time no index was added so he could have added one after the fact. I would drop that index that I provided as it already has one.
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by Jamesonp*:

 

What about deleting the commo rose spam, server spam, blank messages, and optimizing the table after doing so? You should see a pretty huge boost just by doing that.

Already have did that yesterday. Dropped like 10k rows. We have pretty good regex rules set.
* Restored post. It could be that the author is no longer active.
Link to comment

Originally Posted by ty_ger07*:

 

I will be thinking about this...

I am still a little puzzled while adding up the numbers and comparing costs and benefits. If fetching 20 rows takes 4 times longer, it makes me wonder if the hassle and inconvenience (to both me and the user) of overhauling the pagination design is worth it for very little comparable gain. What is a 20 second load time compared to 25 second load time? Both are awful numbers and feel awful for the user. Is it worth hampering the functionality for the user who will be less than impressed by the load time either way?

* Restored post. It could be that the author is no longer active.
Link to comment

Archived

This topic is now archived and is closed to further replies.




  • Our picks

    • Game Server Hosting:

      We're happy to announce that EZRCON will branch out into the game server provider scene. This is a big step for us so please having patience if something doesn't go right in this area. Now, what makes us different compared to other providers? Well, we're going with the idea of having a scaleable server hosting and providing more control in how you set up your server. For example, in Minecraft, you have the ability to control how many CPU cores you wish your server to have access to, how much RAM you want to use, how much disk space you want to use. This type of control can't be offered in a single service package so you're able to configure a custom package the way you want it.

      You can see all the available games here. Currently, we have the following games available.

      Valheim (From $1.50 USD)


      Rust (From $3.20 USD)


      Minecraft (Basic) (From $4.00 USD)


      Call of Duty 4X (From $7.00 USD)


      OpenTTD (From $4.00 USD)


      Squad (From $9.00 USD)


      Insurgency: Sandstorm (From $6.40 USD)


      Changes to US-East:

      Starting in January 2022, we will be moving to a different provider that has better support, better infrastructure, and better connectivity. We've noticed that the connection/routes to this location are not ideal and it's been hard getting support to correct this. Our contract for our two servers ends in March/April respectively. If you currently have servers in this location you will be migrated over to the new provider. We'll have more details when the time comes closer to January. The new location for this change will be based out of Atlanta, GA. If you have any questions/concerns please open a ticket and we'll do our best to answer them.
      • 5 replies
    • Hello All,

      I wanted to give an update to how EZRCON is doing. As of today we have 56 active customers using the services offered. I'm glad its doing so well and it hasn't been 1 year yet. To those that have services with EZRCON, I hope the service is doing well and if not please let us know so that we can improve it where possible. We've done quite a few changes behind the scenes to improve the performance hopefully. 

      We'll be launching a new location for hosting procon layers in either Los Angeles, USA or Chicago, IL. Still being decided on where the placement should be but these two locations are not set in stone yet. We would like to get feedback on where we should have a new location for hosting the Procon Layers, which you can do by replying to this topic. A poll will be created where people can vote on which location they would like to see.

      We're also looking for some suggestions on what else you would like to see for hosting provider options. So please let us know your thoughts on this matter.
      • 4 replies
    • Added ability to disable the new API check for player country info


      Updated GeoIP database file


      Removed usage sending stats


      Added EZRCON ad banner



      If you are upgrading then you may need to add these two lines to your existing installation in the file procon.cfg. To enable these options just change False to True.

      procon.private.options.UseGeoIpFileOnly False
      procon.private.options.BlockRssFeedNews False



       
      • 2 replies
    • I wanted I let you know that I am starting to build out the foundation for the hosting services that I talked about here. The pricing model I was originally going for wasn't going to be suitable for how I want to build it. So instead I decided to offer each service as it's own product instead of a package deal. In the future, hopefully, I will be able to do this and offer discounts to those that choose it.

      Here is how the pricing is laid out for each service as well as information about each. This is as of 7/12/2020.

      Single MySQL database (up to 30 GB) is $10 USD per month.



      If you go over the 30 GB usage for the database then each additional gigabyte is charged at $0.10 USD each billing cycle. If you're under 30GB you don't need to worry about this.


      Databases are replicated across 3 zones (regions) for redundancy. One (1) on the east coast of the USA, One (1) in Frankfurt, and One (1) in Singapore. Depending on the demand, this would grow to more regions.


      Databases will also be backed up daily and retained for 7 days.




      Procon Layer will be $2 USD per month.


      Each layer will only allow one (1) game server connection. The reason behind this is for performance.


      Each layer will also come with all available plugins installed by default. This is to help facilitate faster deployments and get you up and running quickly.


      Each layer will automatically restart if Procon crashes. 


      Each layer will also automatically restart daily at midnight to make sure it stays in tip-top shape.


      Custom plugins can be installed by submitting a support ticket.




      Battlefield Admin Control Panel (BFACP) will be $5 USD per month


      As I am still working on building version 3 of the software, I will be installing the last version I did. Once I complete version 3 it will automatically be upgraded for you.





      All these services will be managed by me so you don't have to worry about the technical side of things to get up and going.

      If you would like to see how much it would cost for the services, I made a calculator that you can use. It can be found here https://ezrcon.com/calculator.html

       
      • 11 replies
    • I have pushed out a new minor release which updates the geodata pull (flags in the playerlisting). This should be way more accurate now. As always, please let me know if any problems show up.

       
      • 9 replies
×
×
  • Create New...

Important Information

Please review our Terms of Use and Privacy Policy. We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.