[BF4] Stats webpage for XpKiller's Stats Logger Plugin

[ImportBot]

Originally Posted by ty_ger07*:

 

I am not sure if you guys have noticed, but you can do fuzzy date time searches on the chat log search page such as '12:00 yesterday' or 'last month' or '2 weeks ago'. I hope you guys enjoy that feature. If you enter a time, it will search for 10 minutes within that time. If you enter a day with no time, it will show all day. If you enter a week with no time, it will show all week. And so on.

[ImportBot]

Originally Posted by ty_ger07*:

 

It's quite clear that XpKiller's session stats are very different than each player's server stats. I don't know why they are so different. I know this issue has been complained about before, but I guess XpKiller has not responded. :S

I guess if all players' session stats are equally screwed up, they will at least still provide a comparison.

[ImportBot]

Originally Posted by ty_ger07*:

 

It has come to my attention that I have put in nearly 1700 hours of overtime so far this year. Essentially, that means that I am working the equivalent of two full time jobs at once. This is my excuse for my slow updates.

[ImportBot]

Originally Posted by ColColonCleaner*:

 

It has come to my attention that I have put in nearly 1700 hours of overtime so far this year. Essentially, that means that I am working the equivalent of two full time jobs at once. This is my excuse for my slow updates.

Damn son, putting in work! Hope the extra hours are paying off though

[ImportBot]

Originally Posted by Jamesonp*:

 

Good job man. Really appreciate the updates!

[ImportBot]

Originally Posted by thunderje*:

 

cancel this post I figured it out ty

[ImportBot]

Originally Posted by ty_ger07*:

 

Update released!

http://tyger07.github.io/BF4-Server-Stats/

 

Changes:

- Most DLC now included (weapons and vehicles still missing)

- Bugs fixed

- More accurate stats

- May be a little slower (due to increased accuracy)

[ImportBot]

Originally Posted by ty_ger07*:

 

Update released

 

Changes:

- DLC Weapons added

- Vehicle images and stats added (where possible)

- Brighter weapon images

[ImportBot]

Originally Posted by Jamesonp*:

 

Thanks for the update

[ImportBot]

Originally Posted by ty_ger07*:

 

Thanks for the update

Wow, your database has gotten huge. The combined chat page, for example, takes 20+ seconds to load because it has to count 1 million+ rows every time.

 

Have you considered pruning it down a bit? I have been messing around a bit with a stats database cleaner/optimizer which I may release for people like you.

[ImportBot]

Originally Posted by dyn*:

 

Have you considered pruning it down a bit? I have been messing around a bit with a stats database cleaner/optimizer which I may release for people like you.

tbl_chatlog 2.1 GiB

 

Showing rows 15280950 - 15280973 ( 15,280,974 total, Query took 20.0390 sec)

 

That would be mighty helpful! Looking through the chat I do see old server spam messages in there and then the ID_CHAT (COMM ROSE) messages. I'm sure just that stuff takes up a ton of space.

[ImportBot]

Originally Posted by Prophet731*:

 

Wow, your database has gotten huge. The combined chat page, for example, takes 20+ seconds to load because it has to count 1 million+ rows every time.

 

Have you considered pruning it down a bit? I have been messing around a bit with a stats database cleaner/optimizer which I may release for people like you.

Is that for pagination? What is the query that took 20 seconds? Ours is 2.78GB with ~25,821,404 rows. There is a missing index for the logDate column and if your searching on that it will take a while. After I added the index to ours, datetime searching was drastically reduced.

 

Run this query on the database and tell me if it speed it up for you. It might take a while on a large table.

 

Code:

ALTER TABLE `tbl_chatlog` ADD INDEX (logDate);

[ImportBot]

Originally Posted by ty_ger07*:

 

Is that for pagination? What is the query that took 20 seconds? Ours is 2.78GB with ~25,821,404 rows. There is a missing index for the logDate column and if your searching on that it will take a while. After I added the index to ours, datetime searching was drastically reduced.

 

Run this query on the database and tell me if it speed it up for you. It might take a while on a large table.

 

Code:

ALTER TABLE `tbl_chatlog` ADD INDEX (logDate);

Yeah, it's for counting the number of rows for pagination. It's not my database; I hope you are directing your suggestion towards those who need it.

[ImportBot]

Originally Posted by Prophet731*:

 

Yeah, it's for counting the number of rows for pagination. It's not my database; I hope you are directing your suggestion towards those who need it.

Yeah its for everyone.

[ImportBot]

Originally Posted by ty_ger07*:

 

Yeah its for everyone.

Thank you for the tip.

Actually, I was thinking -- after I wrote the reply above -- it might be a very important thing I could include in some optimizer code.

 

Edit: I should open my eyes. Now that I see who you are based on your merits, I should be extra grateful for any guidance you are willing to offer to a self-educated PHP/SQL/HTML/CSS/Javascript hack like me.

[ImportBot]

Originally Posted by Prophet731*:

 

Thank you for the tip.

Actually, I was thinking -- after I wrote the reply above -- it might be a very important thing I could include in some optimizer code.

Only reason I found out the index was missing was when I was trying to trim down my response times with my live scoreboard and chat. Took 2 seconds for the chat to be return when filtering by date to a remote DB. After adding that index its now less than 1 second so huge increase.

 

All about those indexes on the right columns that make the difference

[ImportBot]

Originally Posted by ty_ger07*:

 

tbl_chatlog 2.1 GiB

 

Showing rows 15280950 - 15280973 ( 15,280,974 total, Query took 20.0390 sec)

 

That would be mighty helpful! Looking through the chat I do see old server spam messages in there and then the ID_CHAT (COMM ROSE) messages. I'm sure just that stuff takes up a ton of space.

It's really easy to get rid of the commo rose and server spam. Do you know what phpMyAdmin is? If not: phpMyAdmin is a database administration tool every web server host should provide you a link to somewhere in your account which you can use to log into your database, view and alter your database, and issue queries with.

 

Once logged into phpMyAdmin, issue the following queries:

ID_CHAT.png

Code from screenshot above (removes commo rose messages):

Code:

DELETE
FROM `tbl_chatlog`
WHERE `logMessage` LIKE '%ID_CHAT%'

And to remove Server spam:

Code:

DELETE
FROM `tbl_chatlog`
WHERE `logSoldierName` = 'Server'

And to remove blank messages (probably messages in other languages with Cyrillic characters):

Code:

DELETE
FROM `tbl_chatlog`
WHERE `logMessage` = ''

With a huge database, that will probably remove quite a few lines. You may also consider deleting old chat messages if you don't think you will ever look back that far.

 

To remove chat older than 6 months:

Code:

DELETE
FROM `tbl_chatlog`
WHERE `logDate` < DATE_SUB(CURRENT_DATE(),INTERVAL '182' DAY);

After you are all done, you should optimize the table:

Code:

OPTIMIZE TABLE `tbl_chatlog`

And of course, consider 13lackHawk's advice above.

[ImportBot]

Originally Posted by Prophet731*:

 

ty_ger you're going to get a kick out of this. Out of the ~25 million chat records we have, ~10,464,057 are done by the Server spam. With the commo rose its ~3,990,484. Our server does love to chat don't ya think. haha.

 

Here are the SQL for those that want to see how many entries there are for both of them.

 

Code:

SELECT     FORMAT(COUNT(ID), 0) AS 'Total'
FROM
    tbl_chatlog
WHERE
    `logMessage` LIKE 'ID_CHAT%';

Code:

SELECT 
    FORMAT(COUNT(ID), 0) AS 'Total'
FROM
    tbl_chatlog
WHERE
    `logSoldierName` = 'Server';

[ImportBot]

Originally Posted by Jamesonp*:

 

Is that for pagination? What is the query that took 20 seconds? Ours is 2.78GB with ~25,821,404 rows. There is a missing index for the logDate column and if your searching on that it will take a while. After I added the index to ours, datetime searching was drastically reduced.

 

Run this query on the database and tell me if it speed it up for you. It might take a while on a large table.

 

Code:

ALTER TABLE `tbl_chatlog` ADD INDEX (logDate);

Thanks! I'm not really seeing much of a difference on page creation times. Also my ADK Web Admin site fails to load the chatlog for players who have been around for a long time. It stalls and Chrome asks me to kill the page.

 

Any other tips for speeding the database up?

 

@ty_ger07

 

I was going through my slow-query log to try and alleviate some of the slow load times and found the following:

 

# Time: 141027 13:35:06

# User@Host: adkatsro[adkatsro] @ localhost [] Id: 12374

# Schema: adkat_statlog_bf4 Last_errno: 0 Killed: 0

# Query_time: 14.003345 Lock_time: 0.000163 Rows_sent: 3 Rows_examined: 4577048 Rows_affected: 0

# Bytes_sent: 499

SET timestamp=1414442106;

SELECT `logDate`, `logSoldierName`, TRIM(`logMessage`) AS Message, `logSubset`

FROM `tbl_chatlog`

WHERE `ServerID` in (4,5,6,7)

AND `logMessage` != ''

AND `logMessage` NOT LIKE '%ID_CHAT_%'

AND `logSoldierName` != 'Server'

ORDER BY logDate ASC, `logDate` ASC

LIMIT 2216880, 20;

The added 5 seconds to make a 20 second load time is this:

 

# User@Host: adkatsro[adkatsro] @ localhost [] Id: 4831

# Schema: adkat_statlog_bf4 Last_errno: 0 Killed: 0

# Query_time: 5.742400 Lock_time: 0.000150 Rows_sent: 1 Rows_examined: 2360132 Rows_affected: 0

# Bytes_sent: 66

use adkat_statlog_bf4;

SET timestamp=1414441655;

SELECT count(`logDate`) AS count

FROM `tbl_chatlog`

WHERE `ServerID` in (4,5,6,7)

AND `logMessage` != ''

AND `logMessage` NOT LIKE '%ID_CHAT_%'

AND `logSoldierName` != 'Server';

Any suggestions, or do I just have too much data?

[ImportBot]

Originally Posted by ty_ger07*:

 

I am not sure if the extra ANDs in the WHERE clause (I mean: AND `logMessage` != '' AND `logMessage` NOT LIKE '%ID_CHAT_%' AND `logSoldierName` != 'Server') slow it down more or not. I suppose they might. You could try removing those ANDs from both queries in chat.php, chat-search.php, and chat-live.php and see if it speeds up some.

 

You coupd also try ", `logMessage` AS Message" instead of ", TRIM(`logMessage`) AS Message". I don't know if TRIM takes a lot of extra time. TRIM just removes any extra spaces before or after a message. But, it shouldn't make much difference since you are only trimming 20 messages (a small number).

 

Otherwise, I think what would help a lot are the queries previously mentioned in order to slim down your chat log. You could probably reduce your chat log size drastically simply by removing the server spam, commo rose commands, and blank messages. Look back a few posts. Don't forget to optimize the table after you are done making changes.

 

The second query (the 5 second one you mentioned) is what counts the total number of rows. I would have thought that the second query should be much slower (20 seconds for example) than the first query (5 seconds for example) since the first query is only gathering data from 20 rows while the second query is gathering data from 2 million + rows (to count the total number of rows).

[ImportBot]

Originally Posted by Ruger*:

 

Thanks for the update and awesome work ty_ger07!

[ImportBot]

Originally Posted by Prophet731*:

 

I am not sure if the extra ANDs in the WHERE clause (I mean: AND `logMessage` != '' AND `logMessage` NOT LIKE '%ID_CHAT_%' AND `logSoldierName` != 'Server') slow it down more or not. I suppose they might. You could try removing those ANDs from both queries in chat.php, chat-search.php, and chat-live.php and see if it speeds up some.

 

You coupd also try ", `logMessage` AS Message" instead of ", TRIM(`logMessage`) AS Message". I don't know if TRIM takes a lot of extra time. TRIM just removes any extra spaces before or after a message. But, it shouldn't make much difference since you are only trimming 20 messages (a small number).

 

Otherwise, I think what would help a lot are the queries previously mentioned in order to slim down your chat log. You could probably reduce your chat log size drastically simply by removing the server spam, commo rose commands, and blank messages. Look back a few posts. Don't forget to optimize the table after you are done making changes.

 

The second query (the 5 second one you mentioned) is what counts the total number of rows. I would have thought that the second query should be much slower (20 seconds for example) than the first query (5 seconds for example) since the first query is only gathering data from 20 rows while the second query is gathering data from 2 million + rows (to count the total number of rows).

The query is fine. It's the logMessage column which causes the slow down.

[ImportBot]

Originally Posted by Jamesonp*:

 

Thanks for the reply, to alleviate some of the issues with too much chat log history I've just set it to allow the last 30 days of chat with the following line:

 

AND logDate > NOW() - INTERVAL 30 DAY

Example:

 

{

$TotalRows_q = @mysqli_query($BF4stats,"

SELECT count(`logDate`) AS count

FROM `tbl_chatlog`

WHERE `ServerID` in ({$ids})

AND logDate > NOW() - INTERVAL 30 DAY

AND `logMessage` != ''

AND `logMessage` NOT LIKE '%ID_CHAT_%'

AND `logSoldierName` != 'Server'

");

Also have you thought about removing the pagination count to gather how many pages total and just have it display only the first 10 pages, then when you click the 10th page it generates the next 10 pages. Just like how Google does it on search results?

[ImportBot]

Originally Posted by ty_ger07*:

 

Is there any reason why you don't want to delete the commo rose ("ID_CHAT...") commands or server spam or empty messages? If you are filtering it to only fetch the last 30 days, is there a reason why you don't want to delete the old messages?

 

Regarding your question:

If you are viewing the newest chat messages on the last page, how do you know which page you are on? If the last page is page 1 and you go back a few pages, are you then on page -2? If I am selecting a row number to start from (offset in query), I need to know valid offsets -- positive integers between 0 and infinity which actually exist in the database. "LIMIT 2216880, 20"

I could make it so the webpage doesn't display the number; but still I feel like I need to know the number.

 

I guess that I could make every query start from page 1 (even if it caused the results to be given in backwards order), but it seems awkward. It would send people the message "enjoy manually scrolling through hundreds of thousands of pages to find the last result".

 

 

I do see a mistake with my leaderboard row count logic for combined stats though. I could definitely optimize fetching that number.

[ImportBot]

Originally Posted by Jamesonp*:

 

Is there any reason why you don't want to delete the commo rose ("ID_CHAT...") commands or server spam or empty messages? If you are filtering it to only fetch the last 30 days, is there a reason why you don't want to delete the old messages?

We use the database in conjunction with Adkats Server Admin. Deleting the chat history would be counter productive to allowing us to maintain a record of all things which have occurred on our servers and the chat history for players. We use this tool to aid us in deciding who to ban and to view people's history.

 

If you are viewing the newest chat messages on the last page, how do you know which page you are on? If the last page is page 1 and you go back a few pages, are you then on page -2? If I am selecting a row number to start from (offset in query), I need to know valid offsets -- positive integers between 0 and infinity which actually exist in the database. "LIMIT 2216880, 20"

I could make it so the webpage doesn't display the number; but still I feel like I need to know the number.

Instead of reading the amount of rows each time, would it be possible to store the value in a table and have it update every 30 minutes (or next page view if over 30 minutes)?

 

A good reference on how something like this would work is how Google calculates it's results. For example, if you Google "test" you'll see the following message:

 

"About 247,000 results" and it only shows you the first 10 pages. Once you click page 10, it shows you the next 4 pages possible (14 pages total) and so on.

 

If you need to search for a specific period of time you'd use the search logic you already have in place.

[ImportBot]

Originally Posted by ty_ger07*:

 

We use the database in conjunction with Adkats Server Admin. Deleting the chat history would be counter productive to allowing us to maintain a record of all things which have occurred on our servers and the chat history for players. We use this tool to aid us in deciding who to ban and to view people's history.

What about deleting the commo rose spam, server spam, blank messages, and optimizing the table after doing so? You should see a pretty huge boost just by doing that.

[ImportBot]

Originally Posted by ty_ger07*:

 

Is that for pagination? What is the query that took 20 seconds? Ours is 2.78GB with ~25,821,404 rows. There is a missing index for the logDate column and if your searching on that it will take a while. After I added the index to ours, datetime searching was drastically reduced.

 

Run this query on the database and tell me if it speed it up for you. It might take a while on a large table.

 

Code:

ALTER TABLE `tbl_chatlog` ADD INDEX (logDate);

Could you please explain this?

logdate_index.png

 

It appears that I now have two equal indexes for logDate. Doing so actually seems to have slowed down the query a bit. Perhaps its because the borrowed database I am using is from a user who already ran your webpage code or something? Or maybe XpKiller made a change to his plugin at some time?

[ImportBot]

Originally Posted by Prophet731*:

 

Could you please explain this?

logdate_index.png

 

It appears that I now have two equal indexes for logDate. Doing so actually seems to have slowed down the query a bit. Perhaps its because the borrowed database I am using is from a user who already ran your webpage code or something? Or maybe XpKiller made a change to his plugin at some time?

That could be. When we ran XpKillers logger for BF4 it re-created the tables for us and so I guess at the time no index was added so he could have added one after the fact. I would drop that index that I provided as it already has one.

[ImportBot]

Originally Posted by Jamesonp*:

 

What about deleting the commo rose spam, server spam, blank messages, and optimizing the table after doing so? You should see a pretty huge boost just by doing that.

Already have did that yesterday. Dropped like 10k rows. We have pretty good regex rules set.

[ImportBot]

Originally Posted by ty_ger07*:

 

I will be thinking about this...

I am still a little puzzled while adding up the numbers and comparing costs and benefits. If fetching 20 rows takes 4 times longer, it makes me wonder if the hassle and inconvenience (to both me and the user) of overhauling the pagination design is worth it for very little comparable gain. What is a 20 second load time compared to 25 second load time? Both are awful numbers and feel awful for the user. Is it worth hampering the functionality for the user who will be less than impressed by the load time either way?