Google Gmail - Less secure apps - Kritisk säkerhetsvarning - IMAP/POP3/SMTP services

If you are using the "Less secure apps" SMTP feature of a Gmail account but does not use an unknown count of months between the requests then Google will block the request to use it once the program you are using makes an attempt.

Then the Gmail account receives warnings about that an attempt blocked it and it disabled the less secure apps feature and it asks if it was the user that made the attempt named like critical security error or so ("Kritisk säkerhetsvarning" in Swedish). But there are no links to re-enable it in the mails. Completely bonkers and very unfriendly.

Note how user-unfriendly this is constructed: It does not warn the user that it will disable the access nor does it warn the user that it has disabled it - because it actually waits for the so called less secure app to make the request and once it does that, then disables the feature.

What the less secure app actually wanted is not noted. It could have been a very important request that was denied access.

Page to re-enable Less secure apps

To re-enable less secure apps you have to visit the following page logged in with the account in question:
https://myaccount.google.com/lesssecureapps

Disabling it is all fine and dandy, but when enabling it then you will get junk warnings in the mail that you shall check your activity.

Postfix re-run

In case the less secure apps service that you tried to use was a Postfix server trying to send mails, then here are some tips to get back on track.

To check what went wrong run:
postqueue -p

Also check:
/var/log/mail (and the archived copies named .1, .gz and so on)

To re-try the sending after enabling the less secure apps on the address above:
postqueue -f

https://www.tech-g.com/2012/07/15/inspecting-postfixs-email-queue/

Attempts to solve this

To solve this you need to use the less secure apps access more often. But if you just happen to have a service that does it very sporadically then this is not the easiest task to accomplish. Having to re-enable the less secure apps each time and re-run the service is not a solution.

Therefore a way to use less secure apps feature more often is needed. A cronjob to periodically do something that keeps the feature on, but not something that changes things on the mail account like read flags on the mails or fills it up with unwanted data in form of sent mails.

Idea - let PHP read the mail through imap_*, either through POP3 or IMAP

The idea here was to make a simple script to let PHP login to the Gmail account to refresh the usage and disconnect. Set it as a cronjob and done.

This failed at first, because PHP: 7.3+ imap_*:s functions are relying on an outdated IMAP client at the time of writing this according to sources online, so Gmail refuses logins from imap_* functions in PHP. Turned out it could be avoided using a parameter and that I used wrong credentials.

The below script logs in to the mailbox, fetches a list of sub folders or a list of mails in the selected folder which takes a while and then logs out.

Install PHP and it's requirements - possibly more is needed to setup:
apt install php php-imap

Touch an empty file for the script, for example in ~/gmail-refresh.php
touch ~/gmail-refresh.php
chmod 700 ~/gmail-refresh.php
chown <your username>:<your username> ~/gmail-refresh.php

Fill the file ~/gmail-refresh.php with the script - change the setup lines to match your needs:
#!/usr/bin/php
<?php

# requirements: apt install php php-imap

# setup - username and password
$username = "username@gmail.com";
$password = "account-password";
# setup - set folder to fetch, set to INBOX first 
# and check below on how to get a list of folders
# as these are localized in the account language
$folder = "INBOX";
# $folder = "[Gmail]/Spam";
# $folder = "[Gmail]/Recycle bin";

# server to connect to
$serverline = "{imap.gmail.com:993/imap/ssl/novalidate-cert}";
# login
$mailbox = imap_open($serverline.$folder, $username, $password)
  or die('Login failed: '.imap_last_error());

# optional - get list of available sub folders, set folder to INBOX for this
# and use the content after the {serverline} as folder name
# $list = imap_list($mailbox, $serverline, "*");
# if (is_array($list)) {
#   foreach ($list as $subfolder) {
#     echo $subfolder."\n";
#   }
# }

# get all mails in this folder, indicate that more is done
# other than just logging in

$mails = imap_search($mailbox, "ALL");
# optional - list all mails, imap_search returns false both on error and no messages
# if (is_array($mails)) {
#  foreach ($mails as $mail) {
#    $headers = imap_headerinfo($mailbox, $mail);
#     echo imap_utf8($headers->fromaddress).': '.imap_utf8($headers->subject)."\n";
#  }
# }
if ($error = imap_last_error()) {
  echo 'Search failed: '.$error."\n";
}

# close the connection
imap_close($mailbox);

?>

Run it with:
php ~/gmail-refresh.php

If you want to save bandwidth and fetch an empty folder but do not know it's name, then you will have to first fetch the folder list, then specify the name of the folder.

To do this fill in the credentials in the script, set the folder to INBOX and un-comment the listing part and comment out imap_search, run it, check for the folder and re-comment the listing part again and replace INBOX with the folder you want to fetch.

Put a line in /etc/crontab to run it the first day of the month at 12:00:
0 12 1 * * root /usr/bin/php /home/<your username>/gmail-refresh.php

Evaluation of the PHP method

Did this with 1 month interval, did not help, got the same error and the setting disabled. Lowering the interval.

Idea - use the SMTP client used to send mails just to test the connection

Sure, but I found no way just to test the connection, the only way to do it was to actually send mails, which could be done with PHP anyway since that actually does not use the imap_* functions but the underlying mail sending client in the operating system. But this will send unnecessary mails, not wanted.

Idea - Use fetchmail to just get some headers periodically

According to sources online is fetchmail refusing to leave mails marked as unread, resulting in that this silent cronjob will actually modify the mailbox contents. Not wanted.

It will also deliver the mails to the local system account. Not wanted.

There was also no way found to just get headers.

Idea - Use getmail to get some headers periodically

The same here as for fetchmail regarding the headers - it has to fetch the whole mails.

But with a difference - it will accept not to touch the unread mail flags.

You can specify an output folder instead of the local user account and you can specify a path in the mailbox where no mails exist provided that you have an empty location and bypass the download.

Installation:
apt-install getmail

Setup folders that has to exist:
mkdir -p ~/.getmail
mkdir -p ~/.getmail/downloaded_mails
mkdir -p ~/.getmail/downloaded_mails/tmp
mkdir -p ~/.getmail/downloaded_mails/new
mkdir -p ~/.getmail/downloaded_mails/cur

Make an empty config file and secure it to read and write only
touch ~/.getmail/getmailrc
chmod 600 ~/.getmail/getmailrc
chown <your system username>:<your-system-username> ~/.getmail/getmailrc

Configuration to put in ~/.getmail/getmailrc
# ~/.getmail/getmailrc
# Configuration file to retrieve messages over secure IMAP
# and send them to procmail

[retriever]
type=SimpleIMAPSSLRetriever
# type = SimplePOP3SSLRetriever
server=imap.gmail.com
# server = pop.gmail.com
# mailboxes, has to end with a comma if only one string
# these labels are in the same language as the account
mailboxes = ("[Gmail]/Spam", )
username = username@gmail.com
password = username-for-account

[destination]
type = Maildir
# note in this folder must tmp, mail and cur folders exist
path = ~/.getmail/downloaded_mails/

[options]
verbose=2
# new messages only
read_all=false

# do not alter messages
delete=false
delete_after=0
delete_bigger_than=0
max_bytes_per_session=0
max_message_size=0
max_messages_per_session=0

# do not alter messages
delivered_to=false
received=false

message_log=~/.getmail/getmail.log
message_log_syslog=false
message_log_verbose=true

To check config:
getmail --dump

To run it (add -q to get quiet output):
getmail

You may want to re-run the chown and chmod lines just to make sure the editor has not changed them:
chmod 600 ~/.getmail/getmailrc
chown <your system username>:<your-system-username> ~/.getmail/getmailrc

Put a line in /etc/crontab to run it the first day of the month at 12:00:
0 12 1 * * <your username in the system> /usr/bin/getmail -q

Note in case you want to try requests more often, do not run it more than with 5 minutes delay or you might get blocked.

Evaluation of getmail method

Tried it by disabling the less secure apps, got this warning:
...
SimpleIMAPSSLRetriever:username@gmail.com@imap.gmail.com:993:
getmailrc: credential/login error ([AUTHENTICATIONFAILED] Invalid credentials (Failure))
  0 messages (0 bytes) retrieved, 0 skipped

Re-enabled it, re-ran getmail, got this as intended:
...
SimpleIMAPSSLRetriever:username@gmail.com@imap.gmail.com:993:
  0 messages (0 bytes) retrieved, 0 skipped

POP3 access - Error [AUTH] Web login required:

When I tried to fetch mails from one Gmail account into another using POP3 I got this error even if less secure apps was enabled for the account to fetch from.

I solved it by visiting https://accounts.google.com/b/0/DisplayUnlockCaptcha and clicked the button there for the account that refused to work.

/var/log/mail - postfix/smtp[xxxxx]: error: open database /etc/postfix/sasl_passwd.db: No such file or directory

The postfix sasl_passwd.db database needs to be created and possibly updated for the sasl_passwd stuff to work.

Otherwise this appears in the /var/log/mail log when it tries to send mail through SMTP and the mail does not get sent:

... postfix/qmgr[...]: <id>: from=<internal sender address@...>, size=..., nrcpt=1 (queue active)
... postfix/smtp[...]: error: open database /etc/postfix/sasl_passwd.db: No such file or directory
... postfix/smtp[...]: warning: hash:/etc/postfix/sasl_passwd is unavailable. open database /etc/postfix/sasl_passwd.db: No such file or directory
... postfix/smtp[...]: warning: hash:/etc/postfix/sasl_passwd lookup error for "<smtp host>"
... postfix/smtp[...]: warning: <id>: smtp_sasl_password_maps lookup error
... postfix/smtp[...]: <id>: to=<recipient@...>, relay=<smtp host>[<ip>]:<port>, delay=..., delays=.../.../.../..., dsn=..., status=deferred (local data error while talking to <smtp host>[<ip>])

The solution for this is to create the database:
cd /etc/postfix
sudo postmap sasl_passwd