PHP encryption, stopped receiving emails via 365

[t4Chippy]

Hi Guys, I have a webshop running thirtybees 1.0.7 and around a month ago i stopped receiving "contact us" emails from my site. 

My email account is with microsoft office 365, and is the same domain as my site. My email was configured with "Use PHP's mail() function". This has been working fine since Christmas when the site went live, but as said, just stopped working on the 4th of May. 

I tried switching the customer services email to a gmail account and it comes through just fine, switch back to my domain email with 365 and it stops again. I contacted microsoft and they said to switch it over to SMTP but that refuses to even connect. 

I've read a few bits on bobs online that prestashop used to have a conflict with office 365 and something to do with presta not being STARTTLS compatable, and something to do with the swiftmailer? could this possibly be the issue with thirtybees too?

 

I'm not the most technically gifted individual but i can get a grasp of things and do have a friend who can can fill in the gaps in my knowledge so i thought I'd reach out to you guys and see if we can get this thing going again with your help.

Thanks in advance!

 

[t4Chippy]

2 hours ago, dynambee said:

The difference between -all and ~all is that -all is a stricter setting. It means that any server that is not included in the SPF data should be automatically refused for delivery. ~all is a bit more flexible in that it means that servers not included in the SPF are allowed to send email on behalf of that domain but they are not officially approved. There are reasons that -all might be preferred over ~all but for general use purposes ~all is probably a better choice. (-all might also explain why your incoming TB messages are vanishing.)

I would also double (or triple) check that the IP in your SPF record is actually the IP of your webserver. No zeros that got entered as the letter O, no mistyped numerals, no commas instead of periods, etc.

Thanks Ian, I've amended the SPF entry for the curly, and copy and pasted the IP address to prevent typo's. Do i have to wait for DNS to update?

Also, just to confirm, this is for me to use the outlook 365 SMTP settings in TB, not PHP?

 

Many thanks

Edited June 6, 2019 by t4Chippy

[dynambee]

1 hour ago, t4Chippy said:

Thanks Ian, I've amended the SPF entry for the curly, and copy and pasted the IP address to prevent typo's. Do i have to wait for DNS to update?

Each DNS entry has a TTL (time to live) setting. If a server has queried your SPF record it may cache it until the TTL expires at which point the server would query the SPF record again. Waiting until the TTL expires is certainly the safest option as you know any servers that have cached your SPF record will re-query the record.

 

1 hour ago, t4Chippy said:

Also, just to confirm, this is for me to use the outlook 365 SMTP settings in TB, not PHP?

SPF affects all emails sent on behalf of a given domain. If a server not included in the SPF record tries to send mail on behalf of a domain there is a higher chance that it will be tagged as spam. In theory if -all is used then an unverified server should not be able to have mail delivered on behalf of a domain. However exactly how strict a mail server is with regards to SPF depends on the individual mail server and how the spam filtering is set up.

I have another idea that I will post in a separate message. It will probably take a few minutes to get written up.

[dynambee]

I have been playing with the mail settings on my development tb 1.0.8 server and have noticed a couple of things.

First is that both SSL and TLS are working on my server. I do not have any Office 365 accounts so I can not test if connectivity there is working or not, but I can say that against my mailcow server I can connect both ways. SSL on 465 or TLS on 587. TLS is faster by quite a bit but I am not sure why. In any case TB itself seems fine with both SSL an TLS connections.

Second, the "Send a test mail" button in the back office does NOT seem to work at all, at least not when SMTP is set up. Even when the "Contact us" form can successfully send emails the back office Advanced Parameters/Email "Send a test mail" fails. @lesley is this a known bug?

Third, when you change the settings for the email server and then click save it seems to save an invalid password over whatever password you had there previously. You must paste your password into the password field each time you change the SMTP settings and save them.

Perhaps with this information you will be able to get things working.

[t4Chippy]

56 minutes ago, dynambee said:

I have been playing with the mail settings on my development tb 1.0.8 server and have noticed a couple of things.

First is that both SSL and TLS are working on my server. I do not have any Office 365 accounts so I can not test if connectivity there is working or not, but I can say that against my mailcow server I can connect both ways. SSL on 465 or TLS on 587. TLS is faster by quite a bit but I am not sure why. In any case TB itself seems fine with both SSL an TLS connections.

Second, the "Send a test mail" button in the back office does NOT seem to work at all, at least not when SMTP is set up. Even when the "Contact us" form can successfully send emails the back office Advanced Parameters/Email "Send a test mail" fails. @lesley is this a known bug?

Third, when you change the settings for the email server and then click save it seems to save an invalid password over whatever password you had there previously. You must paste your password into the password field each time you change the SMTP settings and save them.

Perhaps with this information you will be able to get things working.

Hi Ian, thanks for your perseverance 🙂

Okay, 1) I'm wondering whether 365 is just being fussy and looking for the STARTTLS over TLS? I currently have is set to TLS and 587 🙂

This is in my 365 settings.

2) That's interesting, perhaps i won't use that now then.

3) I have saved all the smtp settings including the password and at least for now the contact us page is coming up with an error.

There is 1 error

1. An error occurred while sending the message.

So, that's actually good news for the minute, at least it's not telling customers it's submitted fine like it used to 🙂 

Edited June 6, 2019 by t4Chippy

[dynambee]

Can you connect to any other mail server using TLS on port 587?

[Factor]

2 hours ago, dynambee said:

Second, the "Send a test mail" button in the back office does NOT seem to work at all, at least not when SMTP is set up. Even when the "Contact us" form can successfully send emails the back office Advanced Parameters/Email "Send a test mail" fails. @lesley is this a known bug?

My test email works.  I am on 1.08.  I use cPanel on PHP 7.1 

Note: @dynambee I liked Mailcow.  I tried MailinaBox too, but they didn't want to keep the system updated

[dynambee]

2 minutes ago, Brent Dacus said:

My test email works.  I am on 1.08.  I use cPanel on PHP 7.1 

That's definitely odd as it doesn't work at all for me. Maybe it's related to mailcow, I can try it with gmail at some point.

 

2 minutes ago, Brent Dacus said:

Note: @dynambee I liked Mailcow.  I tried MailinaBox too, but they didn't want to keep the system updated

I also tried mailinabox but I found it restrictive and very lacking in features. That's pretty much the point of mailinabox (simple system with little to go wrong) but it just wasn't for me. I tried a few others as well but settled on mailcow as being the best balance between features, support, and updates. It works well. I'd actually like to migrate to OpenSRS but they do not offer any sort of push mail (no IMAP idle) so I'm still self-hosting for now.

[Factor]

I don't know if your server is on Cpanel? or not but this video might help.  Start at about 8:57

 

[Factor]

If you do have cpanel make sure you check remote mail

1. Tick the 'Remote Mail Exchanger' option in the Change the Email Routing section near the bottom

2. Click the Save button
   
   

[t4Chippy]

18 hours ago, dynambee said:

Can you connect to any other mail server using TLS on port 587?

okay I just set up my gmail account on the smtp, and changed the customer contact to gmail and the webform is coming up with an error

 

 

Does that mean my server doesn't like that port or TLS maybe?

[t4Chippy]

17 hours ago, Brent Dacus said:

I don't know if your server is on Cpanel? or not but this video might help. 

Start at about 8:57

 

Hi Brent, yes all the office side of things and DNS has already been set up, it's been working flawlessly since christmas but just stopped for some reason 😞

[dynambee]

14 minutes ago, t4Chippy said:

Does that mean my server doesn't like that port or TLS maybe?

This is interesting, your server may not support TLS connections. What host are you using? Did you do the openssl tests that I asked about before? What were the results?

[t4Chippy]

49 minutes ago, dynambee said:

This is interesting, your server may not support TLS connections. What host are you using? Did you do the openssl tests that I asked about before? What were the results?

Sorry about server enquires, I'm heavily reliant on my friend for server side stuff, He's just followed your link and it looks like our server does support TLS connections 🙂 

Host is https://www.eukhost.com/ 

Edited June 7, 2019 by t4Chippy

[dynambee]

Here is some sample PHP code that you should be able to use to send a test message. You (or your friend) will need to customize it with the directory path to the swift_required.php file as well as put the right to/from email addresses in and the correct password. This will create a TLS connection to your server and send the message. Looking at the swiftmailer code it would seem (as best I can tell) that if TLS is specified that it tries STARTTLS first. I am no PHP guru though and the swiftmailer code is beyond my abilities.

Put the sample code in a PHP file on your server somewhere you the webserver can see it, and make sure the permissions are correctly set. Then you can access it from a browser to run the PHP and send the mail. If you put it in the root directory and name the file test_tls.php you would need to do something like this to run the file: http://mydomain.com/test_tls.php

Here is the code:

<?php

// If you don't get the path right the webserver will give you an error and from the error you should be able to deduce the correct path:
require_once '[you will need to put the full path here so that the webserver can find the file] /vendor/swiftmailer/swiftmailer/lib/swift_required.php';


// Create the Transport
$transport = (new Swift_SmtpTransport('smtp.office365.com', 587, 'tls'))
  ->setUsername('your-username-here-its-probably-your-email-address')
  ->setPassword('your-password-here')
  ;


// Create the Mailer using your created Transport
$mailer = new Swift_Mailer($transport);


// Create a message
$message = (new Swift_Message('TLS Test Email'))
  ->setFrom(['youremail@yourdomain.com' => 'Your Name Here'])
  ->setTo(['sendto@sendtodomain.com' => 'Send To Name'])
  ->setBody('Here is the message itself. This is a test of the TLS system.')
  ;


// Send the message, display errors:
    if (!$mailer->send($message, $errors))
    {
        echo "Error:";
        print_r($errors);
    }

?>

 

[dynambee]

@t4Chippy how is this going? Did you get it to work? What was the problem?

[t4Chippy]

17 hours ago, dynambee said:

@t4Chippy how is this going? Did you get it to work? What was the problem?

Hi Ian, just waiting for my friend to try the above, he's got on of those "social life" things unlike me 😄

[t4Chippy]

22 hours ago, dynambee said:

@t4Chippy how is this going? Did you get it to work? What was the problem?

Hi Ian, just got this off my friend 🙂

"Getting a 500 error from that script"

Have PM'd the whole error 🙂

Edited June 9, 2019 by t4Chippy

[dynambee]

Did he correctly modify the script for your server? Did he set up the email addresses and password correctly? The script itself is fine, I wrote it and then tested it before posting it.

Might be better if your friend comes to this thread. It’s hard to talk through a 3rd party.

[dynambee]

Based on further discussion by PM and further testing by myself it would seem somewhat likely that @t4Chippy's webserver IP has been blacklisted by MS or by a 3rd party blacklist provider used by MS.

This could be as a result of the regular connections by PHP Mailer or it might be because the DNS records for the domain did not show the webserver IP as being a valid IP to send mail from. Without the DNS records showing the webserver IP as an approved mail server for the domain it makes that IP appear to be sending spam.

The exact reasons are speculation, but what does seem to be true is that the swiftmailer version included with TB is able to connect to smtp.office365.com using TLS.

At this point @t4Chippy should check if his IP has been blacklisted and use something like this to see if his entire domain is on a blacklist.

If the IP is blacklisted then getting a new IP for the webserver would be a good idea. If the domain itself is blacklisted, that is a much bigger problem. Getting a domain removed from public blacklists is just about impossible. The usual suggestion is to abandon the domain and start over with a new one.

If neither the IP nor the domain appear on public blacklists then (assuming it is a blacklist issue) it is probably just the MS SMTP server blacklisting the IP. In this case a new IP for the webserver would be the solution, combined with proper DNS records of course.

[t4Chippy]

43 minutes ago, dynambee said:

Based on further discussion by PM and further testing by myself it would seem somewhat likely that @t4Chippy's webserver IP has been blacklisted by MS or by a 3rd party blacklist provider used by MS.

This could be as a result of the regular connections by PHP Mailer or it might be because the DNS records for the domain did not show the webserver IP as being a valid IP to send mail from. Without the DNS records showing the webserver IP as an approved mail server for the domain it makes that IP appear to be sending spam.

The exact reasons are speculation, but what does seem to be true is that the swiftmailer version included with TB is able to connect to smtp.office365.com using TLS.

At this point @t4Chippy should check if his IP has been blacklisted and use something like this to see if his entire domain is on a blacklist.

If the IP is blacklisted then getting a new IP for the webserver would be a good idea. If the domain itself is blacklisted, that is a much bigger problem. Getting a domain removed from public blacklists is just about impossible. The usual suggestion is to abandon the domain and start over with a new one.

If neither the IP nor the domain appear on public blacklists then (assuming it is a blacklist issue) it is probably just the MS SMTP server blacklisting the IP. In this case a new IP for the webserver would be the solution, combined with proper DNS records of course.

Okay, I'll try and work through the above list to see whats going on, and I do have an MS engineer getting in touch today so I'll get him to see if there's a MS blacklist I'm on. 

So, I've emailed the MAiltester and it's come back perfect at 10/10, 
"SpamAssassin likes you" and "You're not blacklisted" 

But I have noticed (even though it's ticked) "You're not fully authenticated" and under that is "You do not have a DMARC record" so I'll add that TXT to DNS, and also "Your reverse DNS does not match with your sending domain." 

Reverse DNS lookup or reverse DNS resolution (rDNS) is the determination of a domain name that is associated with a given IP address.
Some companies such as AOL will reject any message sent from a server without rDNS, so you must ensure that you have one.
You cannot associate more than one domain name with a single IP address. 

Mind you that was from Outlook on my PC, had the test email ought to be sent from the server?

 

Blacklist check came back with 1?

 

Edited June 10, 2019 by t4Chippy

[dynambee]

33 minutes ago, t4Chippy said:

Blacklist check came back with 1?

Here is some more info on that blacklist.

You might want to try checking your IP's sender score. There's a good chance they won't have much info on you, but they might.

Good that your domain doesn't seem to be blacklisted, that would really suck. Changing webserver IPs is much easier than changing domains.

[t4Chippy]

42 minutes ago, dynambee said:

Here is some more info on that blacklist.

You might want to try checking your IP's sender score. There's a good chance they won't have much info on you, but they might.

Good that your domain doesn't seem to be blacklisted, that would really suck. Changing webserver IPs is much easier than changing domains.

Okay so Our servers IP's sender score is 9 😞

 

[dynambee]

3 minutes ago, t4Chippy said:

Okay so Our servers IP's sender score is 9 😞

9/100. 😞

[dynambee]

At this point it has been definitively shown that MS is blocking SMTP requests from @t4Chippy's webserver. We don't know why the IP is being blocked but the test script runs perfectly from a server on a different IP using @t4Chippy's email account credentials.

So, in final summary, the swiftmailer implementation included with TB supports STARTTLS and connects just fine to MS Office 365 SMTP server. The issue here was specific to the webserver in question.

The solution will either be to get MS to take the IP off the blacklist (difficult IMO) or to change webserver IPs. Also, DNS settings regarding email authentication need to be fully set up to hopefully avoid this problem happening again on a new IP.

[t4Chippy]

21 minutes ago, dynambee said:

At this point it has been definitively shown that MS is blocking SMTP requests from @t4Chippy's webserver. We don't know why the IP is being blocked but the test script runs perfectly from a server on a different IP using @t4Chippy's email account credentials.

So, in final summary, the swiftmailer implementation included with TB supports STARTTLS and connects just fine to MS Office 365 SMTP server. The issue here was specific to the webserver in question.

The solution will either be to get MS to take the IP off the blacklist (difficult IMO) or to change webserver IPs. Also, DNS settings regarding email authentication need to be fully set up to hopefully avoid this problem happening again on a new IP.

I Have asked the MS technician to verify the above so will report back as soon as i hear anything.