Some nice information on Mass emails
Monday 19th November, 2007Here's a technote about how to handle an issue you might see on Domino when sending mass emails through your servers. Sometimes mass emails are sent by the Administrator to notify everyone; however, sometimes end users have been known to send very large emails to several hundred people and not realize the impact they cause. Then they start a massive chain reaction when people keep choosing "Reply to All" instead of just "Reply".
Technote 1227966:
http://www-1.ibm.com/support/docview.wss?rs=0&q1=1227966&uid=swg21227966
In addition to the technote, a fellow co-worker wrote a nice piece on what all to consider when sending mass emails.
"Thanks for you insight Walter."
When a user sends a mass mail (any mail message with more than 250 recipients), the mail delivery system may be taxed beyond the planned capacity. This unexpected spike in workload can result in poor performance as seen by the end-user's perspective.
These end-users may perceive degraded client response times, longer mail delivery times, and the inability for to log onto the Domino system.
To understand how mass mailing affects server performance, it is helpful to understand the mail routing workload.
When sending any mail message, Domino performs three operations.
Each phase of message delivery has unique performance considerations which we cover below:
Addressing the Message
Addressing the message, also referred to as recipient list expansion or message dispatching, comprises several steps:
Special addressing considerations apply:
The $Users view is required by Notes clients to open a new session, therefore this very brief lock invoked by the router task may delay the opening of new sessions by the Notes clients. Under normal conditions on a healthy server, no response time issues result from this delay. Benchmark testing shows that Domino can deliver a .027 second response time. However, a mass mailing on a heavily load server will likely result in response time issues for end users. Additionally, the size of the $Users and $Groups views affect name resolution performance.
Router attempts to address all messages in mail.box on each sweep of the queue If the number of messages or quantity of recipients is very large, the Domino router task might get delayed in dispatching messages, thus the router will never deliver any mail as no mail has yet been dispatched.
Transferring the Message
If the mail recipients' mail files are not located on the local server, the router task transfers the message elsewhere.
After resolving the home mail server for each recipient for a message, the message transfer thread (within the router task) handles the delivery to the others servers in the domain.
The router task has a number of threads which perform message transfer, as defined in the server configuration document. The amount of processing power and memory used by the transfer threads depends on three factors: the size of the message, number of destination servers, and number of transfer threads.
Router allocates enough memory to keep the entire message in memory for transfer, If a message has a 4 MB attachment, and there are 10 transfer threads, and at least 10 recipients to transfer the message to, router would allocate more than 40 MB of memory for this message. Limiting the number of threads can reduce the amount of memory used by router, but will lengthen the time required to transfer the same number of messages.
Delivering the Message
Delivering the message pertains to writing the message to the recipient's personal mail file. The Domino router task will locally deliver any message from the mail.box which has been dispatched and resolved to the current server if that is the user's home mail server. Router has dedicated threads to perform this operation, the number of which is defined in the configuration document. Like transfer threads, delivery threads must allocate enough memory to store the entire message in memory. The same performance concerns affecting transfer affect local delivery.
The minimize the impact on performance due to sending mass email, do the following.
Do not send mass mail from a production mail server.
Avoid sending mass email. If mass email must be sent, do not send it from a production mail server but an administration server or another dedicated Domino server which is not directly servicing end users. Distributing this workload will prevent end users from experiencing performance issues related to address expansion during phase one of the message delivery process.
This dedicated Domino server should be configured to hold messages for transfer outside of peak system usage.
Reduce the number of threads for transfer and delivery in the Domino router
If you must sent mass mailings, reducing the number of transfer and delivery threads will decrease the CPU and memory requirements.
**Note:** This modification may increase the time required for message delivery.
Limit message dispatch,
The Domino router can configured to dispatch smaller numbers of messages (not recipients) for delivery in the mail.box, thus reducing the peak CPU and memory usage for the Domino router server task. Doing so, however, may increase delivery time.
To configure, utilize the setting: MailDispatchThreshold=xxx . (Where xxx is the number of concurrent messages to dispatch; the recommended setting is 100.)
Limit mass mailing
To configure, utilize the following setting: RouterMaxEffectiveSize=xxx (where xxx is in kilobytes.)
When enabled, this restriction will cause the router to present the Non-Delivery Failure (NDF) message when the number of recipients multiplied by the message size (in k bytes) is greater than the value specified by the notes.ini setting. Attachments are excluded. The value of the RouterMaxEffectiveSize=setting is set in KB.
Implement the following notes.ini setting: RouterMaxEffectiveSizeIncAttach=1.
With this setting enabled, any file attachments are included in the equation to calculate the size of the message.
Increase the capacity of your system to support mass mailing.
Most production Domino mail environments are sized based on the usual end user workloads, Mass mailing, however, is not part of that normal workload. Therefore if mass mailing is part of your business requirements, increase the CPU and memory available to the specific Domino servers that are expected to support this additional workload.
For example, if you wish to configure the Domino router with 25 transfer threads and 25 delivery threads, and the largest expected mass mail item will be 10 MB, sent to 1000 users:
Router will require an additional 500 MB of memory to support this particular mass mailing.
To determine the additional CPU required:
By default, the Lotus Domino server can only support 20 concurrent user requests, (available threads in the pool) Router can service 250% the load of Server, increasing the system requirements by the same is the recommendation.
Example: an Intel 4 way system driving 80% CPU, 30% on the NSERVER task would mean router is expected to require 3 additional CPU to support the load.
Limiting Factors......
Domino is a 31 Bit, 32 bit or 64 bit application depending on the platform, Total server memory is limited by platform from (1.2 Gb to over 4Gb), If the memory requirements of mass mailing combined with the memory requirements of users exceeds the supported memory of your platform, server hangs or crashes may result.
Technote 1227966:
http://www-1.ibm.com/support/docview.wss?rs=0&q1=1227966&uid=swg21227966
In addition to the technote, a fellow co-worker wrote a nice piece on what all to consider when sending mass emails.
"Thanks for you insight Walter."
When a user sends a mass mail (any mail message with more than 250 recipients), the mail delivery system may be taxed beyond the planned capacity. This unexpected spike in workload can result in poor performance as seen by the end-user's perspective.
These end-users may perceive degraded client response times, longer mail delivery times, and the inability for to log onto the Domino system.
To understand how mass mailing affects server performance, it is helpful to understand the mail routing workload.
When sending any mail message, Domino performs three operations.
- Message Addressing: The Domino server reads the To, Cc and Bcc fields, resolves all addressees and then determines which messages are intended for local delivery
- Message Transfer: If one or more recipients have a different home mail server than the sender, the message is queued for transfer to the correct destination mail server(s) based on the available routing tables.
- Message Delivery: The addressed message is transferred from the server's mail.box database to the mail file of the recipient on the local server.
Each phase of message delivery has unique performance considerations which we cover below:
Addressing the Message
Addressing the message, also referred to as recipient list expansion or message dispatching, comprises several steps:
Special addressing considerations apply:
- if the message is in Notes hierarchical name format or in Internet format
- if groups exist (which must be expanded)
- if user names are listed in the Cc and/or Bcc fields
The $Users view is required by Notes clients to open a new session, therefore this very brief lock invoked by the router task may delay the opening of new sessions by the Notes clients. Under normal conditions on a healthy server, no response time issues result from this delay. Benchmark testing shows that Domino can deliver a .027 second response time. However, a mass mailing on a heavily load server will likely result in response time issues for end users. Additionally, the size of the $Users and $Groups views affect name resolution performance.
Router attempts to address all messages in mail.box on each sweep of the queue If the number of messages or quantity of recipients is very large, the Domino router task might get delayed in dispatching messages, thus the router will never deliver any mail as no mail has yet been dispatched.
Transferring the Message
If the mail recipients' mail files are not located on the local server, the router task transfers the message elsewhere.
After resolving the home mail server for each recipient for a message, the message transfer thread (within the router task) handles the delivery to the others servers in the domain.
The router task has a number of threads which perform message transfer, as defined in the server configuration document. The amount of processing power and memory used by the transfer threads depends on three factors: the size of the message, number of destination servers, and number of transfer threads.
Router allocates enough memory to keep the entire message in memory for transfer, If a message has a 4 MB attachment, and there are 10 transfer threads, and at least 10 recipients to transfer the message to, router would allocate more than 40 MB of memory for this message. Limiting the number of threads can reduce the amount of memory used by router, but will lengthen the time required to transfer the same number of messages.
Delivering the Message
Delivering the message pertains to writing the message to the recipient's personal mail file. The Domino router task will locally deliver any message from the mail.box which has been dispatched and resolved to the current server if that is the user's home mail server. Router has dedicated threads to perform this operation, the number of which is defined in the configuration document. Like transfer threads, delivery threads must allocate enough memory to store the entire message in memory. The same performance concerns affecting transfer affect local delivery.
The minimize the impact on performance due to sending mass email, do the following.
Do not send mass mail from a production mail server.
Avoid sending mass email. If mass email must be sent, do not send it from a production mail server but an administration server or another dedicated Domino server which is not directly servicing end users. Distributing this workload will prevent end users from experiencing performance issues related to address expansion during phase one of the message delivery process.
This dedicated Domino server should be configured to hold messages for transfer outside of peak system usage.
Reduce the number of threads for transfer and delivery in the Domino router
If you must sent mass mailings, reducing the number of transfer and delivery threads will decrease the CPU and memory requirements.
**Note:** This modification may increase the time required for message delivery.
Limit message dispatch,
The Domino router can configured to dispatch smaller numbers of messages (not recipients) for delivery in the mail.box, thus reducing the peak CPU and memory usage for the Domino router server task. Doing so, however, may increase delivery time.
To configure, utilize the setting: MailDispatchThreshold=xxx . (Where xxx is the number of concurrent messages to dispatch; the recommended setting is 100.)
Limit mass mailing
To configure, utilize the following setting: RouterMaxEffectiveSize=xxx (where xxx is in kilobytes.)
When enabled, this restriction will cause the router to present the Non-Delivery Failure (NDF) message when the number of recipients multiplied by the message size (in k bytes) is greater than the value specified by the notes.ini setting. Attachments are excluded. The value of the RouterMaxEffectiveSize=setting is set in KB.
Implement the following notes.ini setting: RouterMaxEffectiveSizeIncAttach=1.
With this setting enabled, any file attachments are included in the equation to calculate the size of the message.
Increase the capacity of your system to support mass mailing.
Most production Domino mail environments are sized based on the usual end user workloads, Mass mailing, however, is not part of that normal workload. Therefore if mass mailing is part of your business requirements, increase the CPU and memory available to the specific Domino servers that are expected to support this additional workload.
For example, if you wish to configure the Domino router with 25 transfer threads and 25 delivery threads, and the largest expected mass mail item will be 10 MB, sent to 1000 users:
Router will require an additional 500 MB of memory to support this particular mass mailing.
To determine the additional CPU required:
By default, the Lotus Domino server can only support 20 concurrent user requests, (available threads in the pool) Router can service 250% the load of Server, increasing the system requirements by the same is the recommendation.
Example: an Intel 4 way system driving 80% CPU, 30% on the NSERVER task would mean router is expected to require 3 additional CPU to support the load.
Limiting Factors......
Domino is a 31 Bit, 32 bit or 64 bit application depending on the platform, Total server memory is limited by platform from (1.2 Gb to over 4Gb), If the memory requirements of mass mailing combined with the memory requirements of users exceeds the supported memory of your platform, server hangs or crashes may result.
[0]