A Valuable Lesson Learned
Tags : BlogSphere
Bookmark :
I really need to start actively blogging again,, I've been doing so many other things it kinda took a back seat. To start the process I'm going to admit to something that probably belongs alongside all the stories from Bill and Pauls Worst Practices session at Lotusphere...
One of the new features in BlogSphere V3 Beta 9 was the integration of OpenLog to catch any errors that might occur when the rendering agents ran. I also included the ability to output different levels of debugging information so if anybody contacted me with a problem I could get them to turn on the debug and they would be able to send me logs that would help me trace the problem quickly. There are a couple of different levels of debugging ranging from minimal to insane. When set to 'insane' every stage of the agent produces a new debug document in the database so just viewing your blogs homepage could result in about 200 new debugging documents.
While testing it out I turned on debugging to the 'insane' level on this blog and then promptly forgot about it. I went on vacation to Disney World for a week and came back to find my server starting to run very low on disk space. In over just one week I had managed to add over 7 million debug documents to my blog database increasing it's size to just over 26 Gb.
The Domino server was still ok and serving pages from my blog but obviously I had to delete all these documents from the database. I had a number of options, I could go into the view where they were all stored and try select them all and delete them manually or I could write an agent to delete all the documents.
Have you ever tried a 'Select All' on over 7 million documents, it was still trying to select them after one day. I had to drop my Domino server just to get the notes client to timeout as even ctrl-break wasn't working on the client.
So I decided to try the agent route, it was working very slowly and would delete about 75,000 documents an hour. My guess is that it had to keep recalculating the view's index and it would have taken about 4 days to get rid of the documents and I'd still have to deal with the deletion stubs taking up space.
In the end I copied the .nsf from the server to my client and then deleted the file from the server. I then created a new replica of the database back onto the server and wrote in a replication formula to exclude the debug document. It only took about 18 hours to complete with most of the time taken up by the Notes client reading in the local database to figure out what to replicate and then about 40 minutes for it to actually create the new replica.
In the end I ended up with a new replica of my blog minus the 7 million debug documents and no deletion stubs to worry about.
And the lesson... 'Insane' means insane, if you do turn on debugging in Blogsphere for any reason make sure you remember to turn it off when finished and delete the debug documents.
Bookmark :
I really need to start actively blogging again,, I've been doing so many other things it kinda took a back seat. To start the process I'm going to admit to something that probably belongs alongside all the stories from Bill and Pauls Worst Practices session at Lotusphere...
One of the new features in BlogSphere V3 Beta 9 was the integration of OpenLog to catch any errors that might occur when the rendering agents ran. I also included the ability to output different levels of debugging information so if anybody contacted me with a problem I could get them to turn on the debug and they would be able to send me logs that would help me trace the problem quickly. There are a couple of different levels of debugging ranging from minimal to insane. When set to 'insane' every stage of the agent produces a new debug document in the database so just viewing your blogs homepage could result in about 200 new debugging documents.
While testing it out I turned on debugging to the 'insane' level on this blog and then promptly forgot about it. I went on vacation to Disney World for a week and came back to find my server starting to run very low on disk space. In over just one week I had managed to add over 7 million debug documents to my blog database increasing it's size to just over 26 Gb.
The Domino server was still ok and serving pages from my blog but obviously I had to delete all these documents from the database. I had a number of options, I could go into the view where they were all stored and try select them all and delete them manually or I could write an agent to delete all the documents.
Have you ever tried a 'Select All' on over 7 million documents, it was still trying to select them after one day. I had to drop my Domino server just to get the notes client to timeout as even ctrl-break wasn't working on the client.
So I decided to try the agent route, it was working very slowly and would delete about 75,000 documents an hour. My guess is that it had to keep recalculating the view's index and it would have taken about 4 days to get rid of the documents and I'd still have to deal with the deletion stubs taking up space.
In the end I copied the .nsf from the server to my client and then deleted the file from the server. I then created a new replica of the database back onto the server and wrote in a replication formula to exclude the debug document. It only took about 18 hours to complete with most of the time taken up by the Notes client reading in the local database to figure out what to replicate and then about 40 minutes for it to actually create the new replica.
In the end I ended up with a new replica of my blog minus the 7 million debug documents and no deletion stubs to worry about.
And the lesson... 'Insane' means insane, if you do turn on debugging in Blogsphere for any reason make sure you remember to turn it off when finished and delete the debug documents.
Comments
I have had similar issues, even if I just needed to delete half a million documents. On R5, if you delete a large number of documents (higher than 50K but lower than 100K), the server run out of space when compacting the database... At least that is my experience...
Very annoying to have to delete 50K documents, compact, delete 50K documents, etc...
Posted by Karl-Henry Martinsson At 10:40:43 AM On 11/29/2007 | - Website - |
You can do stubless deletes (jonvon has published this at: { Link }
)
You can disable the views updating by using notesview.AutoUpdate = False
for all views in the db.
Posted by Lars Berntrop-Bos At 09:32:10 AM On 11/30/2007 | - Website - |