Permanently Delete Office 365 Groups

If you have created teams or channels in Microsoft Teams, you likely know this creates Office 365 groups. Many other Microsoft products in the 365/Azure space create Office 365 groups. This is Microsoft’s new group which allows great flexibility across services.

However, if you have ever decided to delete a Sharepoint site or Microsoft Team, you will find you cannot create another team or site in its place. You will receive an error saying this group still exists.

This is because the group was delete as a ‘soft delete’. Meaning it’s sitting in a recycle bin for a number of days until it’s permanently deleted.

You can speed up this process. The easiest way to do this is to connect via Powershell and run the following commands

  1. Launch Powershell
  2. Run the following command if you don’t have AzureAD installed
Install-Module -Name AzureAD
  1. Connect to AzureAD
Connect-AzureAD
  1. Remove deleted groups
Get-AzureADMSDeletedGroup | Remove-AzureADMSDeletedDirectoryObject

If you don’t feel safe running the above command, just run the Get-AzureADMSDeletedGroup first to see what will be removed.

This will take some time to sync.

Storage Spaces 2019 – Slow with parity

I’ve written a lot about Storage Spaces and slow performance. You can find some of my articles here:

We’ve done a lot of work on Storage Spaces recently to try and find out why our new parity array on server 2019 was slow.

The hardware is the following:

  • Lenovo SR650
  • 32GB Memory
  • Xeon CPU
  • 12x 8TB 7200RPM 512e drives

When creating the Storage Space, the logical sector size is set from the disk. These disks are 512e drives with a 512 sector size. The default sector size created is 4k.

You can either buy native 4k drives at the outset, or set your Storage Space to 4k.

Here is our storage spaces before the change:



From my techs

By default when creating a storage pool, the logical sector size should match the highest physical sector in the disk pool, in this case it should be 4K, but maybe due to this drive is 512e drive, so the logical sector stays in 512. This will cause 8 times delay and performance delay due to the OS and disk controller is doing so called “RMW” operation. The picture below described how the performance get affected by this way.

Once we change our Storage Space to 4k, we started to get 120-160mb/s across the array, all the time. Before this, we were getting 30-60mb/s with many stutters.

I hope this helps.

TRIM over iSCSI/NFS

I’ve taken this post from Reddit, which you can find here. The author is BloodyIron.

So I just went down a rabbit hole for about 3hrs, as I NEEDED to know if iSCSI and NFS can pass TRIM over their protocols to SSD-backed storage. And here are my findings.

  1. iSCSI is capable of passing TRIM correctly
  2. NFS requires v4.2 on server and client to pass TRIM correctly, otherwise earlier versions DO NOT

I cobbled this together from an eye-spinning number of sources on the internet. So if you feel you can conclusively prove me wrong, by all means.

I’m primarily posting this for myself (as my blog/website is not yet production ready), and maybe it can help some other people who are looking.

Furthermore:

  1. RHEL 7.4+ has official support for NFS v4.2
  2. FreeBSD, unsure when it will get NFS v4.2, I’m trying to find out, so far I haven’t found info
  3. Proxmox, to get it to use NFS v4.2 (not sure if it can or not) you have to change NFS options for mounts at the CLI, I’ve opened a feature request to add options/settings like this to the webGUI
  4. FreeNAS seems to conclusively not be NFS v4.2 capable, as of this writing (since it also relies on FreeBSD)

Hope this helps someone! 😀

Removing an Office 365 Tenancy

There may be a reason you wish to totally remove an Office 365 tenancy. In our case, it was that the company we looked after was sold. They wanted the data removed – and quickly.

It is possible now to totally remove a tenancy following these steps:

  1. Remove any licensing from the Office 365 tenancy
  2. Open Powershell
  3. Connect to Azure AD by typing
    Connect-AzureAD

    If this doesn’t work, you may need to install AzureAD. Do this by typing

    Install-Module -Name AzureAD
  4. Once connected, you need to connect to Active Directory or mosl. To do this type
    Connect-MsolService

    If this does not work, you may need to install msol. Do this by typing

    Install-Module -Name Connect-MsolService
  5. Disable dirsync with the following command, if enabled
    Set-MsolDirSyncEnabled -EnableDirSync $false

    This command will take around 30 minutes for all users to become in cloud users

  6. You now need to remove all users and remove them from the recycle bin. Type
    Get-MsolUser | Remove-MsolUser -Force

    Then after waiting 30 minutes or so, type the following

    Get-MsolUser -ReturnDeletedUsers | Remove-MsolUser -RemoveFromRecycleBin -Force

    This command removes the deleted users from the AD recycle bin

  7. The next script will remove all of the enterprise applications in AD. This needs to be done
    $ObjectIds = (Get-AzureADServicePrincipal).ObjectIdFor ($i=0; $i -lt $ObjectIds.Length; $i++){ Remove-AzureADServicePrincipal -objectid $ObjectIds[$i]}
  8. Once these commands are completed, you can check Azure Active Directory by going to https://aad.portal.azure.com. Select Azure Active Directory and try to delete it. You will get something like the following. In this case, once the licenses have expired (these we removed 12 hours ago) you will be able to delete the tenancy.

For more information check out the following links:

RemoteApp Slow after Windows10 1803 update on Server 2012 R2

As the title suggests, after updating Windows 10 computers to 1803, users have reported slow RemoteApp sessions.

You can try disabling Remote FX, but user reports suggest this causes further issues.

The easiest fix is to copy mstsc.exe and mstscax.dll from a 1709 build and replace the files on 1803. We have confirmed this works.

KB4103727 Breaks RDP/Remote Desktop Gateway

This morning we awoke to screams from users not being able to login to our remote desktop servers.

KB4103727 has been released which switches a flag to protect against the CredSSP attack.

The quickest way to fix this to get your users working is to patch your domain controller with the May updates and use group policy to push out a change

You can manually add this to the registry for desktop clients

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\CredSSP\Parameters] "AllowEncryptionOracle"=dword:00000002

or via command line

reg add "HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\CredSSP\Parameters" /f /v AllowEncryptionOracle /t REG_DWORD /d 2

To fix this problem, the May updates need to be installed on all servers and workstations.

More information:

 

Unifi Upgrade – MongoDB does not start

Recently had an issue where upgrading Unifi resulted in MongoDB not starting. Looking to the /var/log/unifi/mongod.log we see:

ERROR: Insufficient free space for journal files
[initandlisten] Please make at least 3379MB available in /usr/lib/unifi/data/db/journal or use --smallfiles
[initandlisten]
[initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
[initandlisten] dbexit:
[initandlisten] shutdown: going to close listening sockets...
[initandlisten] shutdown: going to flush diaglog...
[initandlisten] shutdown: going to close sockets...
[initandlisten] shutdown: waiting for fs preallocator...
[initandlisten] shutdown: lock for final commit...
[initandlisten] shutdown: final commit...
[initandlisten] shutdown: closing all files...
[initandlisten] closeAllFiles() finished
[initandlisten] journalCleanup...
[initandlisten] removeJournalFiles
[initandlisten] shutdown: removing fs lock...
[initandlisten] dbexit: really exiting now

This is caused by the disk space being too low (under 3.3GB). Taking a look at the filesystem, we can see this is the case:

The type service unifi restart and you should be good to go.

One issue I found from this is that the MongoDB is increasing at a rapid rate. I found an excellent script here which prunes the content in MongoDB for Unifi Controllers.

Unifi 5.6.22 Upgrade Fails to Start Linux ERROR system – [exec] error, rc=255

When upgrading Unifi to 5.6.22, you will find that Unifi does not start on Linux system.

If you see the log files, you will see something like this:

/var/log/unifi/system.log

 <UniFi> ERROR system - [exec] error, rc=255

This is because of a change in the release notes:

The controller will not start if it is set to bind to a privileged port (<1024), as it now runs as a non-root user

To fix this, you must change your config file to use a port outside of the 1024 range. This will be an issue if you’ve changed your port, like me.

  1. Edit /var/lib/unifi/system.properties
  2. Change your ports from
    unifi.http.port=80
    unifi.https.port=443

    to

    unifi.http.port=8080
    unifi.https.port=8443
  3. Now type
    sudo service unifi restart

You should now be able to login to your system using the new ports.