HDS Stops recording data in SQL dadabase

13 Posts
5 Users
0 Likes
86 Views
(@e.ecochard@arcinfo.com)
Posts: 0
New Member Guest
 

Hello,

For no apparent reason data stop being recorded in the SQL database.

Environment: Windows 8 - Multi-station with 2 redundant real-time servers and 2 redundant Historical servers (multiple active).

Yesterday the recording stopped on both servers at approximately the same time. When looking at the application and logs files, the data to record are correctly sent to the HDS (by VAR).
Following DL advise, I activated the "writing" HDS logs and got the following message:

"CSchTable::AbleToWrite on 'TRENDTABLE1' is FALSE. ID='2'"

That message could explain the the issue but I am not sure of the meaning and don't know why PcVue wouldn't be "AbleToWrite" in the database.

Doesn't anyone knows the meaning of that message?

Moreover, I also see a lot of the following messages:
"Start UrgentShrink on database FLOWDATA"
"Start shrink on database FLOWDATA"
"Start shrink second chance on database FLOWDATA"
"End shrink on database FLOWDATA"
"End UrgentShrink on database FLOWDATA"

Thank you.

Emmanuel.

 
Posted : 22/10/2014 11:24 pm
LM
 LM
(@l-micaudarcinfo-com)
Posts: 383
Member Admin
 

Hello Manu,
Database is probably full or almost full.
Urgent Shrink is not good. You should configure a maintenance plan to avoid that.
Probably CPU is 100% active and there is not enough resources to continue archiving.

That are suppositions regarding what you wrote.

 
Posted : 22/10/2014 11:43 pm
(@r.buisson@arcinfo.com)
Posts: 0
New Member Guest
 

I think that Ludo have the ggod answer.

Have a look on your physical database size and the size put in PcVue

 
Posted : 23/10/2014 1:49 am
(@e.ecochard@arcinfo.com)
Posts: 0
New Member Guest
 

Hello,

Thanks for the reply.
Here are some additional information:
- I was able to access the server and I did a query of the last 1000 records, no data!! For whatever reason the database has been completely emptied.
When looking at the database files, the MDF file was around 2 MB while the transaction log LDF file was around 300 GB!!!
- Fortunately, the customer had a backup that we restored and we changed the PcVue database maximum size to 600 GB (instead of the default 3.5 GB). Note that once restored the database MDF file was around 2.7 GB and the LDF file around 2 MB.
- After that, went to sleep and guess what? This morning same issue of tags that stopped being recorded for no reason!!
Anyway, I really don't know what the hell is going on but it is starting to be huge issue...Will need a divine intervention very soon...

Best Regards,

Emmanuel

 
Posted : 24/10/2014 1:10 am
n.kunzer
(@n-kunzerarcinfo-com)
Posts: 1236
Member Moderator
 

Manu, I suggest you purge the log file in SQL Server.

http://technet.microsoft.com/en-us/library/ms178037%28v=sql.105%29.aspx
http://stackoverflow.com/questions/56628/how-do-you-clear-the-sql-server-transaction-log

I also suggest you contact Arnaud and explain your issue. Maybe he experienced this issue already...

 
Posted : 24/10/2014 6:38 am
n.kunzer
(@n-kunzerarcinfo-com)
Posts: 1236
Member Moderator
 

Ayo, sorry I just realized I misunderstood. I thought the LDF file size was 2.7Gb!!!

 
Posted : 24/10/2014 6:46 am
LM
 LM
(@l-micaudarcinfo-com)
Posts: 383
Member Admin
 

Manu did you check the options of the database in SQL Server Management Studio?

Recovery model should be set to 'Simple' in the 'Options' tab of the properties.
This way the transaction file should not at all grow.

 
Posted : 24/10/2014 12:46 pm
(@admin_doc72)
Posts: 493
Member Admin
 

It is the typical thing that happens and that has repeatedly been critisized and complained about (please remember our discussion during the ITM 2014!).

What happens is that customers go with the default settings for the database setup (3.5GB or any other limitation higher than that). After a certain amount of time the limit is exceeded and PcVue starts to urgently shrink. I emphasize: PcVue does the shrink, not the SQL Server! How does PcVue do it? It deletes recordset by recordset until the configured amount of data has been discarded, instead of using the SQL Server's builtin features. *

On SQL Server side this results in a literal explosion of the log file size because every single DELETE-operation is persisted in the log file, if you go with the default recovery model settings (which is 'Full Backup').
Now, to fully complete the circle of pain, here comes another fact: the limit that you apply in the HDS settings for the database size (3.5GB by default) is the limit for the complete size of the database (that means: data file size plus tranaction log file size). Consequently, we find ourselves in an infinite loop of purging data because the log file grows in at least the same size that the data file is shrinked.

One more thing: In Emmanuel's case the size of the log file is so big for a second reason, if I understand well. Both historical servers try to write into the same database. This is not good. What happens is that both server will write a record. That means SQL Server writes two INSERT entries to the log file. Both records have same timestamp, variable and value. That results in a primary key violation and one of the two records will be discarded. Simplified spoken that means three database transactions that will be logged for one change of value, instead of only one. That means triple the size of the transaction log!

Conclusions:
- Never, never, never, never, never, never, never leave the HDS configuration by default settings!
- Do not let PcVue do the urgent shrink!
- If possible, implement your own backup solution and do not count on 'Full Backup' on SQL Server side. It does not serve our needs because it also includes DELETE operations. [EDIT:] SQL Server native replication seems to require 'Full-Backup' recovery mode (see comment below).
- Always use maintainance plans for shrink and backup!
- Never let two historical servers write to the same database in multiple-active-server-mode!
- If you are nevertheless using contingent purge and you are using 'Full-Backup' recovery it is preferrable to use the SQL Server builtin purge mechanism which potentially results in a smaller transaction log.

[EDIT:]
*) During my investigations about this topic I discovered that the value of the MaxPurgedRecords settings in the HDSConf.dat can be set to 0, which results in using the SQL Server builtin purge mechanism. I have modified the text above accordingly.

 
Posted : 24/10/2014 12:56 pm
(@r.buisson@arcinfo.com)
Posts: 0
New Member Guest
 

Thank you Armin for you reply, you avoid me to do it 😉

 
Posted : 24/10/2014 12:59 pm
n.kunzer
(@n-kunzerarcinfo-com)
Posts: 1236
Member Moderator
 

Armin, I would like reacting on 1 of your recommendations:

You said:
- Implement your own backup solution and do not count on 'Full Backup' on SQL Server side. It does not serve our needs because it also includes DELETE operations.

But, if i am not mistaken, the 'Full backup' option is mandatory if you are using the SQL replication...

 
Posted : 03/11/2014 10:09 am
f.martin
(@f-martinarcinfo-com)
Posts: 148
Member Admin
 

Hello,

Late answer. As I'm writing Release Notes I find a lots of interesting information...

Which version are you using ? This kind of issue has been partially fixed in version 10.0SP2 and version 11.0

 
Posted : 04/11/2014 9:15 pm
(@admin_doc72)
Posts: 493
Member Admin
 

But, if i am not mistaken, the 'Full backup' option is mandatory if you are using the SQL replication...

Okay, that is good to know. I did not know. I am going to update my previous post accordingly.

This kind of issue has been partially fixed in version 10.0SP2 and version 11.0

I have experienced the described behviour with 10.0SP1 and reproduced it again with 11.1 beta. Of what kind of fixes are you exactly speaking about?

For your information: Meanwhile I have created an SPR for the apparent problem of the database size not being calculated correctly for urgent shrink. The number is #60379.

 
Posted : 06/11/2014 2:44 pm
(@e.ecochard@arcinfo.com)
Posts: 0
New Member Guest
 

Hello,

Dominique L. and I spent time doing test on the customer project an finally figured out the issue of lack of recording. Dominique was actually already reworking the way the sampling was done ((with the option "Trend recording mode - Force one record per period" enabled) and the new sampling mechanism he developed will fix that issue.

Thank you all for your participation and the good information provided.
Thank you Armin for doing the SPR about the database size issue in case of urgent shrinking.

FYI, I created a SPR regarding the HDS:
- 60416: SQL Databse Maximum size property can't be set to 0 (to indicate no size limit)

Emmanuel.

 
Posted : 06/11/2014 9:10 pm