2018.2 Adaxes Service High Memory Usage

Question

Since upgrading to 2018.2 we are experiencing an issue where the Softerra.Adaxes.Service will consume all of the OS's available memory causing it to become unresponsive. If I try to restart the service, it will fail and stop. I will start the service and it will work for a little while normally, but then the issue will reoccur. The below message keeps repeating in the log every 5 to 10 minutes or so:

Internal event: The LDAP server has exceeded the limit of the LDAP Maximum Result Set Size. A stored Result Set will be discarded. This will result in a client being unable to continue a paged LDAP search.

I'm attempting to disable scheduled tasks one by one to see if any one is potentially causing the issue, but is there any way to increase the logging level for the service or figure out what might be causing the high memory utilization?

Comments

Hello,

Since upgrading to 2018.2 we are experiencing an issue where the Softerra.Adaxes.Service will consume all of the OS's available memory causing it to become unresponsive. If I try to restart the service, it will fail and stop. I will start the service and it will work for a little while normally, but then the issue will reoccur.

Could you post here or send us (support[at]adaxes.com) the following details:

• The length of Adaxes Command Queue. To check it:

  1. Get the backend port number of your Adaxes service. To do this:
     • In Adaxes Administration Console, right-click your service node.
     • Click Properties in the context menu.
     • The backend port will be displayed below the Check for updates button.
  2. Log on to the computer where Adaxes is installed.
  3. Stop Softerra Adaxes Service.
  4. Launch the LDP tool. To do this:
     • Press Win+R.
     • Type ldp.exe.
     • Press Enter.
  5. In the LDP tool window, click Connection and then click Connect.
     
  6. In the Server field specify the fully qualified domain name of Adaxes server.
  7. In the Port field, specify the service port you noted on step 1 and click OK.
     
  8. Click Connection and then click Bind.
     
  9. Specify the username, password and domain of the Adaxes service account (the account used to run Adaxes Service).
  10. Make sure that the Bind with credentials and Encrypt traffic after bind options are enabled and click OK.
      
  11. In the View menu, select Tree.
  12. In the Base DN drop-down list, select CN=Local Partition,CN={GUID}.
      
  13. Click OK.
  14. Navigate to CN=Local Partition,CN={GUID} \ CN=Adaxes ... \ CN=Pipeline Processor ... .
  15. Right-click CN=Command Queue ... and click Search.
  16. In the Filter field of the Search dialog, enter (objectClass=*).
  17. Select One level and click Run.
      
  18. In the Result Pane located to the right, check how many entries were fetched.

• The actual size of the AdaxesLog.db3 file. By default, the file is located in folder C:\Program Files\Softerra\Adaxes 3\Service\LogDB\ on the computer where Adaxes service is installed.

is there any way to increase the logging level for the service or figure out what might be causing the high memory utilization?

There is no such possibility.

---

Below is the information you have requested:

---

Hello,

Could you check General log for operations executed when the memory gets low? For information on how to view General log, have a look at the following help article: https://www.adaxes.com/help/?Logging.Vi ... eLogl.html. Just refresh the log via F5 during the memory consumption period and note the operations displayed in the log. Then disable the objects that perform the operations and check the memory.

---

The memory consumption appears to correlate directly with the following scheduled task (the 1 hour difference is due to DST). Below is the script, do you see any issues with this? I've disabled it for now to see if the issue returns.

$o365Credential = $Context.GetOffice365Credential()
$session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $o365Credential -Authentication Basic -AllowRedirection
Import-PSSession $session

Add-MailboxPermission "%displayName%" -User "company administrator" -AccessRights 'FullAccess'

Remove-PSsession $session   

Copy code

---

Hello,

We recommend you to use the following script instead of the one you currently have:

try
{
    # Get the object ID in Office 365
    $objectId = [Guid]$Context.TargetObject.Get("adm-O365ObjectId")
}
catch
{
    return # The user doesn't have an Office 365 account
}

try
{
    # Connect to Exchange Online
    $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" `
        -Credential $Context.GetOffice365Credential() -Authentication Basic -AllowRedirection
    Import-PSSession $session -AllowClobber -DisableNameChecking -CommandName "Add-MailboxPermission"

    # Change mailbox type
    Add-MailboxPermission $objectId.ToString() -User "company administrator" -AccessRights 'FullAccess'
}
finally
{
    # Close the remote session and release resources
    if ($session) { Remove-PSSession $session }
}

ExpandCopy code

It is a common practice to use try/finally when establishing sessions. If you have other scripts with the same approach, we recommend you to modify them to use try/finally too.

---

The issue has not returned so I think that confirms this task was the root cause. I've tried the updated version of the script that you posted, but I'm running into throttling issues on the O365 side, any techniques to avoid this?

---

Hello,

This happens because there is a new connection being established for each user for which the Scheduled Task is executed. Could you provide us with all the possible details regarding what exactly you need the script to do? It might be possible to set the permissions using the built-in Modify Exchange properties action without using a script.

---

Basically the script's sole purpose is to add full access for the company administrator to all mailboxes. Ideally you would only execute the script on mailboxes that do not already have the permission, but in my experience, doing a get-mailbox and then add-mailbox only exacerbates the throttling issue. To your point though, it seems we would be better off running this over DomainDNS instead of User so it can utilize the same session for all users?

---

Hello,

This might be not a good solution as the script will probably take a lot of time to run. For us to suggest a solution that will meet your needs, could you clarify whether the company administrator user exists in your on-premise Active Directory?

---

No it doesn't, this is a built-in group in exchange online/O365 that contains all O365 global admins.

Answer 1

Hello,

Thank you for clarifying. We recommend the following solution:

1. Update permissions for all existing mailboxes in Windows PowerShell using the below script. When prompted, enter the credentials of the account that was used to register your Office 365 tenant in Adaxes.

try
{
    # Connect to Exchange Online
    $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" `
        -Credential (Get-Credential) -Authentication Basic -AllowRedirection
    Import-PSSession $session -AllowClobber -DisableNameChecking -CommandName "Add-MailboxPermission", "Get-Mailbox"

    # Change mailbox type
    Get-Mailbox -ResultSize unlimited | Add-MailboxPermission -User "company administrator" -AccessRights 'FullAccess'
}
finally
{
    # Close the remote session and release resources
    if ($session) { Remove-PSSession $session }
}

Copy code

2. Use the Scheduled Task to update the permissions only for new users. You will need to mark the users by setting a specific property value. For example, you can set an Adaxes custom Boolean attribute (e.g. CustomAttributeBoolean1) to True after assigning Office 365 licenses in a Business Rule triggering After Creating a User. The Scheduled Task will run the script only for users that have the attribute set to true and have a mailbox. Also, the Scheduled Task will clear the attribute.
   The Business Rule will look like the following:
   
   The Scheduled Task in its turn will look like the following and use the below script:
   

try
{
    # Get the object ID in Office 365
    $objectId = [Guid]$Context.TargetObject.Get("adm-O365ObjectId")
}
catch
{
    return # The user doesn't have an Office 365 account
}

try
{
    # Connect to Exchange Online
    $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" `
        -Credential $Context.GetOffice365Credential() -Authentication Basic -AllowRedirection
    Import-PSSession $session -AllowClobber -DisableNameChecking -CommandName "Add-MailboxPermission"

    # Change mailbox type
    Add-MailboxPermission $objectId.ToString() -User "company administrator" -AccessRights 'FullAccess'
}
finally
{
    # Close the remote session and release resources
    if ($session) { Remove-PSSession $session }
}

ExpandCopy code