unable to detect any running worker

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

unable to detect any running worker

Colin
Hi,

For a while now I've had stability problems with ASSP. It has generally
been one or two restarts a day. When I upgraded to 16270 I had huge
problems with delayed mail, mail not getting through at all and ASSP
continually shutting down each time there were thousands of "unable to
detect any running worker" spewed into the log files. Yesterday I dropped
back to 16256 and at least mail is flowing but so far I've had one ASSP
instance shut itself off 8 times today.

I've two instances doing this, both on Ubuntu 14.04.

Last night I set debugging on and caught the incident on both servers
within half an hour. I've looked through the debug file but there is
nothing I can see to indicate any errors. Both servers were handling mail
from different senders in the few minutes leading up to the fault.

So I looked back through previous threads on the same issue today and saw a
Thomas ask what the worker status page showed when it happens. I was
wondering how on earth I was going to catch it when it happens and before
it reboots then lo and behold whilst I was on the web interface I saw the
errors flying past in a tail of the maillog.

I went on the web interface and the dot at the top had turned red. I then
went on the worker status page and that was all green.

Up until now, I have been running 10 workers which is possibly overkill. I
had just reduced this instance of ASSP to 5 workers as a test. Status of
the workers is:
1,2,3,5 - ThreadGetNewCon with loop age 0s (worker 3 had 1s)
4 - Maillog
10000 - MonitorMainThread (0s)
10001 - schedule waiting (71s)

I went back to the main page and the dot had gone back to green but the
maillog was still filling with the running worker errors.

I refreshed the status and the only changes were:
4 - "wh:0 - write: - wait: 0.005"
The time on schedule waiting went up to 96s.

Shortly after ASSP Shut down. It is like the main thread and the workers
just stop talking to each other.

I'd love to crack this and give the latest development version a go because
right now I have the annoying issue of an SSL session taking so long that
the sending server starts sending it again (This is smtproutes.com not
gmail in this case). I've also seen this from Mandrill and got in touch
with their support. They explained that it was down to shared spools. One
server behind the infrastructure picks up the message and starts delivering
it. 10 minutes later it is still there so another server picks it up and
starts delivering it. Whichever completes first removes the file and the
other servers terminate.

All the best,
Colin.

------------------------------------------------------------------------------

_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
So I'm following up on this as no matter what version I run I seem to come
up against this.

I've started noticed a few things.

Firstly, if I send message by SMTP using thunderbird the server can sit
there for a while (10, 20 seconds) at connected before it actually does
anything. It then accepts the message really quickly and then sits there
for a while again.

I then spotted something in the logs:

2016-10-03 08:09:38 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_5 in 31.754 seconds - got (ok)
2016-10-03 08:43:38 [Main_Thread] Info: Main_Thread freed by idle Worker_2
in 31.700 seconds - got (ok)
2016-10-03 08:44:32 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_5 in 31.757 seconds - got (ok)
2016-10-03 08:49:23 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_9 in 31.792 seconds - got (ok)
2016-10-03 08:50:01 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_7 in 31.622 seconds - got (ok)
2016-10-03 08:50:41 [Main_Thread] Info: Main_Thread freed by idle Worker_2
in 31.514 seconds - got (ok)
2016-10-03 08:53:12 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_9 in 29.114 seconds - got (ok)
2016-10-03 08:56:32 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_3 in 31.499 seconds - got (ok)
2016-10-03 08:58:18 [Main_Thread] Info: Main_Thread freed by idle Worker_10
in 31.365 seconds - got (ok)
2016-10-03 08:58:52 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_9 in 31.988 seconds - got (ok)
2016-10-03 09:22:05 [Main_Thread] Info: Main_Thread freed by idle Worker_1
in 31.164 seconds - got (ok)
2016-10-03 09:22:39 [Main_Thread] Info: Main_Thread freed by idle Worker_6
in 30.599 seconds - got (ok)

I actually spotted the idle workers first. Surely if they are idle they
shouldn't take 30 seconds to pick up a connection from the Main Thread?
Then whilst looking at the logs to see what happened to the thread before I
found the interrupted Workers as well. I actually dropped this server back
to 7 workers during the day yesterday but that doesn't seem to have made a
difference.

The big thing here is that they are mostly around 30s. There are some
quicker and some longer. These happen throughout the day whilst I am seeing
the no running worker errors - for example yesterday these were logged
between 08:02 and 18:22.

The first shut down from no running workers was 09:32 and the last 18:25.

If I examine the first one, Worker 5 I read that as 08:09:38 - 32s 08:09:06

At 08:09:05 Worker 5 I see:

2016-10-03 08:09:05 [Worker_5] Info: Worker_5 got connection from MainThread

This is a 29.38kB email from gmail. It finishes this message at 08:09:19

2016-10-03 08:09:19 [Worker_5] Disconnected: session:7FEDCB20ACD8
74.125.82.50 - command list was 'EHLO,STARTTLS,EHLO,MAIL FROM,RCPT
TO,DATA,QUIT' - used 30 SocketCalls - processing time 14 seconds - damped 0
seconds

There is nothing from Worker 5 until:

2016-10-03 08:09:38 [Worker_5] Info: Worker_5 got connection from MainThread
2016-10-03 08:09:38 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_5 in 31.754 seconds - got (ok)

If I go back up in the logs, all previous connections to Worker 5 have been
completed and closed.

So I can't make sense of this. The connection at 08:09:05 is passed off to
Worker_5 so surely the MainThread should be freed straight away. It is
almost like the Workers aren't freeing MainThread and MainThread is timing
them out after 30 seconds.

>From my understanding that would explain why Main_Thread can't accept any
more connections and explain a lot of the delays that people have
complained about.

Have I understood correctly or have I missed something obvious?


On Fri, Sep 30, 2016 at 3:25 PM, cw <[hidden email]> wrote:

> Hi,
>
> For a while now I've had stability problems with ASSP. It has generally
> been one or two restarts a day. When I upgraded to 16270 I had huge
> problems with delayed mail, mail not getting through at all and ASSP
> continually shutting down each time there were thousands of "unable to
> detect any running worker" spewed into the log files. Yesterday I dropped
> back to 16256 and at least mail is flowing but so far I've had one ASSP
> instance shut itself off 8 times today.
>
> I've two instances doing this, both on Ubuntu 14.04.
>
> Last night I set debugging on and caught the incident on both servers
> within half an hour. I've looked through the debug file but there is
> nothing I can see to indicate any errors. Both servers were handling mail
> from different senders in the few minutes leading up to the fault.
>
> So I looked back through previous threads on the same issue today and saw
> a Thomas ask what the worker status page showed when it happens. I was
> wondering how on earth I was going to catch it when it happens and before
> it reboots then lo and behold whilst I was on the web interface I saw the
> errors flying past in a tail of the maillog.
>
> I went on the web interface and the dot at the top had turned red. I then
> went on the worker status page and that was all green.
>
> Up until now, I have been running 10 workers which is possibly overkill. I
> had just reduced this instance of ASSP to 5 workers as a test. Status of
> the workers is:
> 1,2,3,5 - ThreadGetNewCon with loop age 0s (worker 3 had 1s)
> 4 - Maillog
> 10000 - MonitorMainThread (0s)
> 10001 - schedule waiting (71s)
>
> I went back to the main page and the dot had gone back to green but the
> maillog was still filling with the running worker errors.
>
> I refreshed the status and the only changes were:
> 4 - "wh:0 - write: - wait: 0.005"
> The time on schedule waiting went up to 96s.
>
> Shortly after ASSP Shut down. It is like the main thread and the workers
> just stop talking to each other.
>
> I'd love to crack this and give the latest development version a go
> because right now I have the annoying issue of an SSL session taking so
> long that the sending server starts sending it again (This is
> smtproutes.com not gmail in this case). I've also seen this from Mandrill
> and got in touch with their support. They explained that it was down to
> shared spools. One server behind the infrastructure picks up the message
> and starts delivering it. 10 minutes later it is still there so another
> server picks it up and starts delivering it. Whichever completes first
> removes the file and the other servers terminate.
>
> All the best,
> Colin.
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
Further development on this today, very little.
I have moved both servers onto Ubuntu 16.04 LTS which means going from perl
5.18 to 5.22 and rebuilding all perl modules from scratch.

The admin user db did not work after the upgrade so I had to empty the
tables before it would come back online.

I'm still getting delayed emails and assp shutting down telling me it is unable
to detect any running worker.

If this goes on much longer the MD will pull the plug and we'll end up
moving to a third party solution which is not something I want but if I
can't fix it I can't defend it :/

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

K Post
I've been reading here, but I haven't had anything to suggest.  All seems
quite odd if it was working prior to upgrading and downgrading didn't work.


Could you spin up a backup of the installation after copying the current
data?  Sure you'd have an older corpus, but I'd think you could add the new
files if necessary, manually replace whitelist etc.


On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:

> Further development on this today, very little.
> I have moved both servers onto Ubuntu 16.04 LTS which means going from perl
> 5.18 to 5.22 and rebuilding all perl modules from scratch.
>
> The admin user db did not work after the upgrade so I had to empty the
> tables before it would come back online.
>
> I'm still getting delayed emails and assp shutting down telling me it is
> unable
> to detect any running worker.
>
> If this goes on much longer the MD will pull the plug and we'll end up
> moving to a third party solution which is not something I want but if I
> can't fix it I can't defend it :/
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
Cheers for the reply.

I don't think backups are an option seen as I've moved completely from
Ubuntu 14.04 to Ubuntu 16.04.

Also, this issue has been around for months. It was causing a handful of
shutdowns a week with the occasional spat of more frequent shutdowns. It is
entirely possible that the errors are behaviour related and nothing to do
with the upgrade and that current email behaviour is triggering it big
style.

I'm not convinced though, unfortunately I'm not convinced of anything else
hence not having much to go on. I can't see any consistencies in the
behaviour leading up to the events.

I don't think it is database related. It happened on one of the mail
servers during the upgrade to 16.04 when ASSP had just started but I had
not yet got into the web interface to replace the corrupted encrypted
strings with the correct values so all database connections were in error.

The problem has already started again this morning so I can see this being
another fun day that either leads to a fix or having to put something else
in place.

On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:

> I've been reading here, but I haven't had anything to suggest.  All seems
> quite odd if it was working prior to upgrading and downgrading didn't work.
>
>
> Could you spin up a backup of the installation after copying the current
> data?  Sure you'd have an older corpus, but I'd think you could add the new
> files if necessary, manually replace whitelist etc.
>
>
> On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
>
> > Further development on this today, very little.
> > I have moved both servers onto Ubuntu 16.04 LTS which means going from
> perl
> > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> >
> > The admin user db did not work after the upgrade so I had to empty the
> > tables before it would come back online.
> >
> > I'm still getting delayed emails and assp shutting down telling me it is
> > unable
> > to detect any running worker.
> >
> > If this goes on much longer the MD will pull the plug and we'll end up
> > moving to a third party solution which is not something I want but if I
> > can't fix it I can't defend it :/
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Thomas Eckardt/eck
>to replace the corrupted encrypted
strings with the correct values

Was the 'Crypt::GOST' module from the SF download page at the old assp
instance?
If it was, did you install the 'Crypt::GOST' module from the SF download
page, before you started the new assp instance?

https://sourceforge.net/projects/assp/files/ASSP%20V2%20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/

Thomas





Von:    cw <[hidden email]>
An:     ASSP development mailing list <[hidden email]>
Datum:  05.10.2016 09:52
Betreff:        Re: [Assp-test] unable to detect any running worker



Cheers for the reply.

I don't think backups are an option seen as I've moved completely from
Ubuntu 14.04 to Ubuntu 16.04.

Also, this issue has been around for months. It was causing a handful of
shutdowns a week with the occasional spat of more frequent shutdowns. It
is
entirely possible that the errors are behaviour related and nothing to do
with the upgrade and that current email behaviour is triggering it big
style.

I'm not convinced though, unfortunately I'm not convinced of anything else
hence not having much to go on. I can't see any consistencies in the
behaviour leading up to the events.

I don't think it is database related. It happened on one of the mail
servers during the upgrade to 16.04 when ASSP had just started but I had
not yet got into the web interface to replace the corrupted encrypted
strings with the correct values so all database connections were in error.

The problem has already started again this morning so I can see this being
another fun day that either leads to a fix or having to put something else
in place.

On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:

> I've been reading here, but I haven't had anything to suggest.  All
seems
> quite odd if it was working prior to upgrading and downgrading didn't
work.
>
>
> Could you spin up a backup of the installation after copying the current
> data?  Sure you'd have an older corpus, but I'd think you could add the
new

> files if necessary, manually replace whitelist etc.
>
>
> On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
>
> > Further development on this today, very little.
> > I have moved both servers onto Ubuntu 16.04 LTS which means going from
> perl
> > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> >
> > The admin user db did not work after the upgrade so I had to empty the
> > tables before it would come back online.
> >
> > I'm still getting delayed emails and assp shutting down telling me it
is
> > unable
> > to detect any running worker.
> >
> > If this goes on much longer the MD will pull the plug and we'll end up
> > moving to a third party solution which is not something I want but if
I

> > can't fix it I can't defend it :/
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
Hi Thomas,

Thanks for chipping in. All modules are installed by running the latest
mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it requires
a minor edit to the code to get it to build. So that module was installed
by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
make clean
perl Makefile.PL
make
make test
make install

I run cpan-outdated -p|cpanm from time to time in order to keep modules up
to date as well so things don't stay stuck on old versions.

Everything that is installed now is a completely fresh build as the Upgrade
to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had to
be installed from scratch.

I bypassed the issue by truncating the tables and setting up the users
again. With there only being one user it was easier to do that than muck
about with it - especially with the other bigger issue at hand.

On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt <[hidden email]>
wrote:

> >to replace the corrupted encrypted
> strings with the correct values
>
> Was the 'Crypt::GOST' module from the SF download page at the old assp
> instance?
> If it was, did you install the 'Crypt::GOST' module from the SF download
> page, before you started the new assp instance?
>
> <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
> 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>
> Thomas
>
>
>
>
>
> Von:    cw <[hidden email]>
> An:     ASSP development mailing list <[hidden email]>
> Datum:  05.10.2016 09:52
> Betreff:        Re: [Assp-test] unable to detect any running worker
>
>
>
> Cheers for the reply.
>
> I don't think backups are an option seen as I've moved completely from
> Ubuntu 14.04 to Ubuntu 16.04.
>
> Also, this issue has been around for months. It was causing a handful of
> shutdowns a week with the occasional spat of more frequent shutdowns. It
> is
> entirely possible that the errors are behaviour related and nothing to do
> with the upgrade and that current email behaviour is triggering it big
> style.
>
> I'm not convinced though, unfortunately I'm not convinced of anything else
> hence not having much to go on. I can't see any consistencies in the
> behaviour leading up to the events.
>
> I don't think it is database related. It happened on one of the mail
> servers during the upgrade to 16.04 when ASSP had just started but I had
> not yet got into the web interface to replace the corrupted encrypted
> strings with the correct values so all database connections were in error.
>
> The problem has already started again this morning so I can see this being
> another fun day that either leads to a fix or having to put something else
> in place.
>
> On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
>
> > I've been reading here, but I haven't had anything to suggest.  All
> seems
> > quite odd if it was working prior to upgrading and downgrading didn't
> work.
> >
> >
> > Could you spin up a backup of the installation after copying the current
> > data?  Sure you'd have an older corpus, but I'd think you could add the
> new
> > files if necessary, manually replace whitelist etc.
> >
> >
> > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
> >
> > > Further development on this today, very little.
> > > I have moved both servers onto Ubuntu 16.04 LTS which means going from
> > perl
> > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> > >
> > > The admin user db did not work after the upgrade so I had to empty the
> > > tables before it would come back online.
> > >
> > > I'm still getting delayed emails and assp shutting down telling me it
> is
> > > unable
> > > to detect any running worker.
> > >
> > > If this goes on much longer the MD will pull the plug and we'll end up
> > > moving to a third party solution which is not something I want but if
> I
> > > can't fix it I can't defend it :/
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Thomas Eckardt/eck
Colin, do the following,

set:

useDB4IntCache - OFF
WorkerLog - diagnostic
shutdown assp
clean the folder assp/tmpDB
remove all griplist.* files from the assp root folder
check the assp/lib/CorrectASSPcfg.pm - there should be no lines for
'$main::lockBDB' and '$main::lockDatabases' in sub set
start assp

Make sure, your init.d script is waiting until the assp perl process is
finished at a stop request!
Finaly kill the perl process only, if 'MaxFinConWaitTime' + some (eg.5)
seconds is reached!

Thomas


Von:    cw <[hidden email]>
An:     ASSP development mailing list <[hidden email]>
Datum:  05.10.2016 10:10
Betreff:        Re: [Assp-test] unable to detect any running worker



Hi Thomas,

Thanks for chipping in. All modules are installed by running the latest
mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it requires
a minor edit to the code to get it to build. So that module was installed
by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
make clean
perl Makefile.PL
make
make test
make install

I run cpan-outdated -p|cpanm from time to time in order to keep modules up
to date as well so things don't stay stuck on old versions.

Everything that is installed now is a completely fresh build as the
Upgrade
to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had to
be installed from scratch.

I bypassed the issue by truncating the tables and setting up the users
again. With there only being one user it was easier to do that than muck
about with it - especially with the other bigger issue at hand.

On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
<[hidden email]>
wrote:

> >to replace the corrupted encrypted
> strings with the correct values
>
> Was the 'Crypt::GOST' module from the SF download page at the old assp
> instance?
> If it was, did you install the 'Crypt::GOST' module from the SF download
> page, before you started the new assp instance?
>
> <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
> 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>
> Thomas
>
>
>
>
>
> Von:    cw <[hidden email]>
> An:     ASSP development mailing list <[hidden email]>
> Datum:  05.10.2016 09:52
> Betreff:        Re: [Assp-test] unable to detect any running worker
>
>
>
> Cheers for the reply.
>
> I don't think backups are an option seen as I've moved completely from
> Ubuntu 14.04 to Ubuntu 16.04.
>
> Also, this issue has been around for months. It was causing a handful of
> shutdowns a week with the occasional spat of more frequent shutdowns. It
> is
> entirely possible that the errors are behaviour related and nothing to
do
> with the upgrade and that current email behaviour is triggering it big
> style.
>
> I'm not convinced though, unfortunately I'm not convinced of anything
else
> hence not having much to go on. I can't see any consistencies in the
> behaviour leading up to the events.
>
> I don't think it is database related. It happened on one of the mail
> servers during the upgrade to 16.04 when ASSP had just started but I had
> not yet got into the web interface to replace the corrupted encrypted
> strings with the correct values so all database connections were in
error.
>
> The problem has already started again this morning so I can see this
being
> another fun day that either leads to a fix or having to put something
else

> in place.
>
> On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
>
> > I've been reading here, but I haven't had anything to suggest.  All
> seems
> > quite odd if it was working prior to upgrading and downgrading didn't
> work.
> >
> >
> > Could you spin up a backup of the installation after copying the
current
> > data?  Sure you'd have an older corpus, but I'd think you could add
the
> new
> > files if necessary, manually replace whitelist etc.
> >
> >
> > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
> >
> > > Further development on this today, very little.
> > > I have moved both servers onto Ubuntu 16.04 LTS which means going
from
> > perl
> > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> > >
> > > The admin user db did not work after the upgrade so I had to empty
the
> > > tables before it would come back online.
> > >
> > > I'm still getting delayed emails and assp shutting down telling me
it
> is
> > > unable
> > > to detect any running worker.
> > >
> > > If this goes on much longer the MD will pull the plug and we'll end
up
> > > moving to a third party solution which is not something I want but
if

> I
> > > can't fix it I can't defend it :/
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential,
legally
> privileged and protected in law and are intended solely for the use of
the

>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Thomas Eckardt/eck
In reply to this post by Colin
I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This version
shows some more information, if 'WorkerLog' is set to diagnostic.

Thomas





Von:    cw <[hidden email]>
An:     ASSP development mailing list <[hidden email]>
Datum:  05.10.2016 10:10
Betreff:        Re: [Assp-test] unable to detect any running worker



Hi Thomas,

Thanks for chipping in. All modules are installed by running the latest
mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it requires
a minor edit to the code to get it to build. So that module was installed
by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
make clean
perl Makefile.PL
make
make test
make install

I run cpan-outdated -p|cpanm from time to time in order to keep modules up
to date as well so things don't stay stuck on old versions.

Everything that is installed now is a completely fresh build as the
Upgrade
to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had to
be installed from scratch.

I bypassed the issue by truncating the tables and setting up the users
again. With there only being one user it was easier to do that than muck
about with it - especially with the other bigger issue at hand.

On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
<[hidden email]>
wrote:

> >to replace the corrupted encrypted
> strings with the correct values
>
> Was the 'Crypt::GOST' module from the SF download page at the old assp
> instance?
> If it was, did you install the 'Crypt::GOST' module from the SF download
> page, before you started the new assp instance?
>
> <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
> 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>
> Thomas
>
>
>
>
>
> Von:    cw <[hidden email]>
> An:     ASSP development mailing list <[hidden email]>
> Datum:  05.10.2016 09:52
> Betreff:        Re: [Assp-test] unable to detect any running worker
>
>
>
> Cheers for the reply.
>
> I don't think backups are an option seen as I've moved completely from
> Ubuntu 14.04 to Ubuntu 16.04.
>
> Also, this issue has been around for months. It was causing a handful of
> shutdowns a week with the occasional spat of more frequent shutdowns. It
> is
> entirely possible that the errors are behaviour related and nothing to
do
> with the upgrade and that current email behaviour is triggering it big
> style.
>
> I'm not convinced though, unfortunately I'm not convinced of anything
else
> hence not having much to go on. I can't see any consistencies in the
> behaviour leading up to the events.
>
> I don't think it is database related. It happened on one of the mail
> servers during the upgrade to 16.04 when ASSP had just started but I had
> not yet got into the web interface to replace the corrupted encrypted
> strings with the correct values so all database connections were in
error.
>
> The problem has already started again this morning so I can see this
being
> another fun day that either leads to a fix or having to put something
else

> in place.
>
> On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
>
> > I've been reading here, but I haven't had anything to suggest.  All
> seems
> > quite odd if it was working prior to upgrading and downgrading didn't
> work.
> >
> >
> > Could you spin up a backup of the installation after copying the
current
> > data?  Sure you'd have an older corpus, but I'd think you could add
the
> new
> > files if necessary, manually replace whitelist etc.
> >
> >
> > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
> >
> > > Further development on this today, very little.
> > > I have moved both servers onto Ubuntu 16.04 LTS which means going
from
> > perl
> > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> > >
> > > The admin user db did not work after the upgrade so I had to empty
the
> > > tables before it would come back online.
> > >
> > > I'm still getting delayed emails and assp shutting down telling me
it
> is
> > > unable
> > > to detect any running worker.
> > >
> > > If this goes on much longer the MD will pull the plug and we'll end
up
> > > moving to a third party solution which is not something I want but
if

> I
> > > can't fix it I can't defend it :/
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential,
legally
> privileged and protected in law and are intended solely for the use of
the

>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
Thank you Thomas.

useDB4IntCache - already set to off
I've set WorkerLog to diagnostic and done the other steps.
I don't have anything in CorrectASSPcfg.pm so as part of the
troubleshooting I have previously deleted it and downloaded a fresh copy
from SourceForge.

I have to say that ASSP started up within seconds after clearing those
files out - it normally takes several minutes and has always done.

I looked at SF but only see 16275 in test (updated 3 days ago) so maybe it
hasn't made its way live yet.

On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt <[hidden email]>
wrote:

> I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This version
> shows some more information, if 'WorkerLog' is set to diagnostic.
>
> Thomas
>
>
>
>
>
> Von:    cw <[hidden email]>
> An:     ASSP development mailing list <[hidden email]>
> Datum:  05.10.2016 10:10
> Betreff:        Re: [Assp-test] unable to detect any running worker
>
>
>
> Hi Thomas,
>
> Thanks for chipping in. All modules are installed by running the latest
> mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it requires
> a minor edit to the code to get it to build. So that module was installed
> by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
> make clean
> perl Makefile.PL
> make
> make test
> make install
>
> I run cpan-outdated -p|cpanm from time to time in order to keep modules up
> to date as well so things don't stay stuck on old versions.
>
> Everything that is installed now is a completely fresh build as the
> Upgrade
> to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had to
> be installed from scratch.
>
> I bypassed the issue by truncating the tables and setting up the users
> again. With there only being one user it was easier to do that than muck
> about with it - especially with the other bigger issue at hand.
>
> On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
> <[hidden email]>
> wrote:
>
> > >to replace the corrupted encrypted
> > strings with the correct values
> >
> > Was the 'Crypt::GOST' module from the SF download page at the old assp
> > instance?
> > If it was, did you install the 'Crypt::GOST' module from the SF download
> > page, before you started the new assp instance?
> >
> > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
> > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:    cw <[hidden email]>
> > An:     ASSP development mailing list <[hidden email]>
> > Datum:  05.10.2016 09:52
> > Betreff:        Re: [Assp-test] unable to detect any running worker
> >
> >
> >
> > Cheers for the reply.
> >
> > I don't think backups are an option seen as I've moved completely from
> > Ubuntu 14.04 to Ubuntu 16.04.
> >
> > Also, this issue has been around for months. It was causing a handful of
> > shutdowns a week with the occasional spat of more frequent shutdowns. It
> > is
> > entirely possible that the errors are behaviour related and nothing to
> do
> > with the upgrade and that current email behaviour is triggering it big
> > style.
> >
> > I'm not convinced though, unfortunately I'm not convinced of anything
> else
> > hence not having much to go on. I can't see any consistencies in the
> > behaviour leading up to the events.
> >
> > I don't think it is database related. It happened on one of the mail
> > servers during the upgrade to 16.04 when ASSP had just started but I had
> > not yet got into the web interface to replace the corrupted encrypted
> > strings with the correct values so all database connections were in
> error.
> >
> > The problem has already started again this morning so I can see this
> being
> > another fun day that either leads to a fix or having to put something
> else
> > in place.
> >
> > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
> >
> > > I've been reading here, but I haven't had anything to suggest.  All
> > seems
> > > quite odd if it was working prior to upgrading and downgrading didn't
> > work.
> > >
> > >
> > > Could you spin up a backup of the installation after copying the
> current
> > > data?  Sure you'd have an older corpus, but I'd think you could add
> the
> > new
> > > files if necessary, manually replace whitelist etc.
> > >
> > >
> > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
> > >
> > > > Further development on this today, very little.
> > > > I have moved both servers onto Ubuntu 16.04 LTS which means going
> from
> > > perl
> > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> > > >
> > > > The admin user db did not work after the upgrade so I had to empty
> the
> > > > tables before it would come back online.
> > > >
> > > > I'm still getting delayed emails and assp shutting down telling me
> it
> > is
> > > > unable
> > > > to detect any running worker.
> > > >
> > > > If this goes on much longer the MD will pull the plug and we'll end
> up
> > > > moving to a third party solution which is not something I want but
> if
> > I
> > > > can't fix it I can't defend it :/
> > > >
> > > > ------------------------------------------------------------
> > > > ------------------
> > > > Check out the vibrant tech community on one of the world's most
> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > _______________________________________________
> > > > Assp-test mailing list
> > > > [hidden email]
> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >
> > > >
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > *******************************************************
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > *******************************************************
> >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
I missed the bit about init.d.

I haven't touched that for years. The stop command runs:

/usr/local/assp stop
It then removes the pid file if the stop command returns an exit code of 0

I never use the init.d restart command. ASSP shuts itself down and does not
start back up automatically. Nothing external is involved in the shutdown
either, I do have an external script that will start ASSP back up when it
stops but that will take a minimum of 5 minutes to kick in to be sure it
isn't a temporary error or a shut down command is still running.

On Wed, Oct 5, 2016 at 10:47 AM, cw <[hidden email]> wrote:

> Thank you Thomas.
>
> useDB4IntCache - already set to off
> I've set WorkerLog to diagnostic and done the other steps.
> I don't have anything in CorrectASSPcfg.pm so as part of the
> troubleshooting I have previously deleted it and downloaded a fresh copy
> from SourceForge.
>
> I have to say that ASSP started up within seconds after clearing those
> files out - it normally takes several minutes and has always done.
>
> I looked at SF but only see 16275 in test (updated 3 days ago) so maybe it
> hasn't made its way live yet.
>
> On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt <
> [hidden email]> wrote:
>
>> I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This version
>> shows some more information, if 'WorkerLog' is set to diagnostic.
>>
>> Thomas
>>
>>
>>
>>
>>
>> Von:    cw <[hidden email]>
>> An:     ASSP development mailing list <[hidden email]>
>> Datum:  05.10.2016 10:10
>> Betreff:        Re: [Assp-test] unable to detect any running worker
>>
>>
>>
>> Hi Thomas,
>>
>> Thanks for chipping in. All modules are installed by running the latest
>> mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
>> requires
>> a minor edit to the code to get it to build. So that module was installed
>> by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
>> make clean
>> perl Makefile.PL
>> make
>> make test
>> make install
>>
>> I run cpan-outdated -p|cpanm from time to time in order to keep modules up
>> to date as well so things don't stay stuck on old versions.
>>
>> Everything that is installed now is a completely fresh build as the
>> Upgrade
>> to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had to
>> be installed from scratch.
>>
>> I bypassed the issue by truncating the tables and setting up the users
>> again. With there only being one user it was easier to do that than muck
>> about with it - especially with the other bigger issue at hand.
>>
>> On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
>> <[hidden email]>
>> wrote:
>>
>> > >to replace the corrupted encrypted
>> > strings with the correct values
>> >
>> > Was the 'Crypt::GOST' module from the SF download page at the old assp
>> > instance?
>> > If it was, did you install the 'Crypt::GOST' module from the SF download
>> > page, before you started the new assp instance?
>> >
>> > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
>> > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> >
>> > Von:    cw <[hidden email]>
>> > An:     ASSP development mailing list <[hidden email]>
>> > Datum:  05.10.2016 09:52
>> > Betreff:        Re: [Assp-test] unable to detect any running worker
>> >
>> >
>> >
>> > Cheers for the reply.
>> >
>> > I don't think backups are an option seen as I've moved completely from
>> > Ubuntu 14.04 to Ubuntu 16.04.
>> >
>> > Also, this issue has been around for months. It was causing a handful of
>> > shutdowns a week with the occasional spat of more frequent shutdowns. It
>> > is
>> > entirely possible that the errors are behaviour related and nothing to
>> do
>> > with the upgrade and that current email behaviour is triggering it big
>> > style.
>> >
>> > I'm not convinced though, unfortunately I'm not convinced of anything
>> else
>> > hence not having much to go on. I can't see any consistencies in the
>> > behaviour leading up to the events.
>> >
>> > I don't think it is database related. It happened on one of the mail
>> > servers during the upgrade to 16.04 when ASSP had just started but I had
>> > not yet got into the web interface to replace the corrupted encrypted
>> > strings with the correct values so all database connections were in
>> error.
>> >
>> > The problem has already started again this morning so I can see this
>> being
>> > another fun day that either leads to a fix or having to put something
>> else
>> > in place.
>> >
>> > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
>> >
>> > > I've been reading here, but I haven't had anything to suggest.  All
>> > seems
>> > > quite odd if it was working prior to upgrading and downgrading didn't
>> > work.
>> > >
>> > >
>> > > Could you spin up a backup of the installation after copying the
>> current
>> > > data?  Sure you'd have an older corpus, but I'd think you could add
>> the
>> > new
>> > > files if necessary, manually replace whitelist etc.
>> > >
>> > >
>> > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
>> > >
>> > > > Further development on this today, very little.
>> > > > I have moved both servers onto Ubuntu 16.04 LTS which means going
>> from
>> > > perl
>> > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
>> > > >
>> > > > The admin user db did not work after the upgrade so I had to empty
>> the
>> > > > tables before it would come back online.
>> > > >
>> > > > I'm still getting delayed emails and assp shutting down telling me
>> it
>> > is
>> > > > unable
>> > > > to detect any running worker.
>> > > >
>> > > > If this goes on much longer the MD will pull the plug and we'll end
>> up
>> > > > moving to a third party solution which is not something I want but
>> if
>> > I
>> > > > can't fix it I can't defend it :/
>> > > >
>> > > > ------------------------------------------------------------
>> > > > ------------------
>> > > > Check out the vibrant tech community on one of the world's most
>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > _______________________________________________
>> > > > Assp-test mailing list
>> > > > [hidden email]
>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > >
>> > > >
>> > >
>> > > ------------------------------------------------------------
>> > > ------------------
>> > > Check out the vibrant tech community on one of the world's most
>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > _______________________________________________
>> > > Assp-test mailing list
>> > > [hidden email]
>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > >
>> > >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> >
>> >
>> > DISCLAIMER:
>> > *******************************************************
>> > This email and any files transmitted with it may be confidential,
>> legally
>> > privileged and protected in law and are intended solely for the use of
>> the
>> >
>> > individual to whom it is addressed.
>> > This email was multiple times scanned for viruses. There should be no
>> > known virus in this email!
>> > *******************************************************
>> >
>> >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential, legally
>> privileged and protected in law and are intended solely for the use of the
>>
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Thomas Eckardt/eck
In reply to this post by Colin
>I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
it
hasn't made its way live yet.

Sorry, my background CVS sync was not running - update is done.

Thomas




Von:    cw <[hidden email]>
An:     ASSP development mailing list <[hidden email]>
Datum:  05.10.2016 11:48
Betreff:        Re: [Assp-test] unable to detect any running worker



Thank you Thomas.

useDB4IntCache - already set to off
I've set WorkerLog to diagnostic and done the other steps.
I don't have anything in CorrectASSPcfg.pm so as part of the
troubleshooting I have previously deleted it and downloaded a fresh copy
from SourceForge.

I have to say that ASSP started up within seconds after clearing those
files out - it normally takes several minutes and has always done.

I looked at SF but only see 16275 in test (updated 3 days ago) so maybe it
hasn't made its way live yet.

On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
<[hidden email]>
wrote:

> I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
version

> shows some more information, if 'WorkerLog' is set to diagnostic.
>
> Thomas
>
>
>
>
>
> Von:    cw <[hidden email]>
> An:     ASSP development mailing list <[hidden email]>
> Datum:  05.10.2016 10:10
> Betreff:        Re: [Assp-test] unable to detect any running worker
>
>
>
> Hi Thomas,
>
> Thanks for chipping in. All modules are installed by running the latest
> mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
requires
> a minor edit to the code to get it to build. So that module was
installed
> by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
> make clean
> perl Makefile.PL
> make
> make test
> make install
>
> I run cpan-outdated -p|cpanm from time to time in order to keep modules
up
> to date as well so things don't stay stuck on old versions.
>
> Everything that is installed now is a completely fresh build as the
> Upgrade
> to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had
to

> be installed from scratch.
>
> I bypassed the issue by truncating the tables and setting up the users
> again. With there only being one user it was easier to do that than muck
> about with it - especially with the other bigger issue at hand.
>
> On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
> <[hidden email]>
> wrote:
>
> > >to replace the corrupted encrypted
> > strings with the correct values
> >
> > Was the 'Crypt::GOST' module from the SF download page at the old assp
> > instance?
> > If it was, did you install the 'Crypt::GOST' module from the SF
download

> > page, before you started the new assp instance?
> >
> > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
> > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:    cw <[hidden email]>
> > An:     ASSP development mailing list
<[hidden email]>

> > Datum:  05.10.2016 09:52
> > Betreff:        Re: [Assp-test] unable to detect any running worker
> >
> >
> >
> > Cheers for the reply.
> >
> > I don't think backups are an option seen as I've moved completely from
> > Ubuntu 14.04 to Ubuntu 16.04.
> >
> > Also, this issue has been around for months. It was causing a handful
of
> > shutdowns a week with the occasional spat of more frequent shutdowns.
It

> > is
> > entirely possible that the errors are behaviour related and nothing to
> do
> > with the upgrade and that current email behaviour is triggering it big
> > style.
> >
> > I'm not convinced though, unfortunately I'm not convinced of anything
> else
> > hence not having much to go on. I can't see any consistencies in the
> > behaviour leading up to the events.
> >
> > I don't think it is database related. It happened on one of the mail
> > servers during the upgrade to 16.04 when ASSP had just started but I
had

> > not yet got into the web interface to replace the corrupted encrypted
> > strings with the correct values so all database connections were in
> error.
> >
> > The problem has already started again this morning so I can see this
> being
> > another fun day that either leads to a fix or having to put something
> else
> > in place.
> >
> > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
> >
> > > I've been reading here, but I haven't had anything to suggest.  All
> > seems
> > > quite odd if it was working prior to upgrading and downgrading
didn't

> > work.
> > >
> > >
> > > Could you spin up a backup of the installation after copying the
> current
> > > data?  Sure you'd have an older corpus, but I'd think you could add
> the
> > new
> > > files if necessary, manually replace whitelist etc.
> > >
> > >
> > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
> > >
> > > > Further development on this today, very little.
> > > > I have moved both servers onto Ubuntu 16.04 LTS which means going
> from
> > > perl
> > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> > > >
> > > > The admin user db did not work after the upgrade so I had to empty
> the
> > > > tables before it would come back online.
> > > >
> > > > I'm still getting delayed emails and assp shutting down telling me
> it
> > is
> > > > unable
> > > > to detect any running worker.
> > > >
> > > > If this goes on much longer the MD will pull the plug and we'll
end

> up
> > > > moving to a third party solution which is not something I want but
> if
> > I
> > > > can't fix it I can't defend it :/
> > > >
> > > > ------------------------------------------------------------
> > > > ------------------
> > > > Check out the vibrant tech community on one of the world's most
> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > _______________________________________________
> > > > Assp-test mailing list
> > > > [hidden email]
> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >
> > > >
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > *******************************************************
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > *******************************************************
> >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential,
legally
> privileged and protected in law and are intended solely for the use of
the

>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Thomas Eckardt/eck
In reply to this post by Colin
>/usr/local/assp stop

There is no such commandline option in assp!

Have a look in to the files in the assp/rc folder.

Good examples are the four youngest files (debian,bsd,redhat).
Waiting for the assp process at finish time is not an option, THIS IS
MANDATORY !!!

If the process is killed immediatly, all BerkeleyDB's will be destroyed.
ASSP will try to repair them at the next start (takes time !!!!), another
impact may be, that it takes a long time to read and wite them - or access
is locked for a long time.

Thomas




Von:    cw <[hidden email]>
An:     ASSP development mailing list <[hidden email]>
Datum:  05.10.2016 11:57
Betreff:        Re: [Assp-test] unable to detect any running worker



I missed the bit about init.d.

I haven't touched that for years. The stop command runs:

/usr/local/assp stop
It then removes the pid file if the stop command returns an exit code of 0

I never use the init.d restart command. ASSP shuts itself down and does
not
start back up automatically. Nothing external is involved in the shutdown
either, I do have an external script that will start ASSP back up when it
stops but that will take a minimum of 5 minutes to kick in to be sure it
isn't a temporary error or a shut down command is still running.

On Wed, Oct 5, 2016 at 10:47 AM, cw <[hidden email]> wrote:

> Thank you Thomas.
>
> useDB4IntCache - already set to off
> I've set WorkerLog to diagnostic and done the other steps.
> I don't have anything in CorrectASSPcfg.pm so as part of the
> troubleshooting I have previously deleted it and downloaded a fresh copy
> from SourceForge.
>
> I have to say that ASSP started up within seconds after clearing those
> files out - it normally takes several minutes and has always done.
>
> I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
it
> hasn't made its way live yet.
>
> On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt <
> [hidden email]> wrote:
>
>> I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
version

>> shows some more information, if 'WorkerLog' is set to diagnostic.
>>
>> Thomas
>>
>>
>>
>>
>>
>> Von:    cw <[hidden email]>
>> An:     ASSP development mailing list <[hidden email]>
>> Datum:  05.10.2016 10:10
>> Betreff:        Re: [Assp-test] unable to detect any running worker
>>
>>
>>
>> Hi Thomas,
>>
>> Thanks for chipping in. All modules are installed by running the latest
>> mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
>> requires
>> a minor edit to the code to get it to build. So that module was
installed
>> by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
>> make clean
>> perl Makefile.PL
>> make
>> make test
>> make install
>>
>> I run cpan-outdated -p|cpanm from time to time in order to keep modules
up
>> to date as well so things don't stay stuck on old versions.
>>
>> Everything that is installed now is a completely fresh build as the
>> Upgrade
>> to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had
to
>> be installed from scratch.
>>
>> I bypassed the issue by truncating the tables and setting up the users
>> again. With there only being one user it was easier to do that than
muck

>> about with it - especially with the other bigger issue at hand.
>>
>> On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
>> <[hidden email]>
>> wrote:
>>
>> > >to replace the corrupted encrypted
>> > strings with the correct values
>> >
>> > Was the 'Crypt::GOST' module from the SF download page at the old
assp
>> > instance?
>> > If it was, did you install the 'Crypt::GOST' module from the SF
download

>> > page, before you started the new assp instance?
>> >
>> > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
>> > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> >
>> > Von:    cw <[hidden email]>
>> > An:     ASSP development mailing list
<[hidden email]>
>> > Datum:  05.10.2016 09:52
>> > Betreff:        Re: [Assp-test] unable to detect any running worker
>> >
>> >
>> >
>> > Cheers for the reply.
>> >
>> > I don't think backups are an option seen as I've moved completely
from
>> > Ubuntu 14.04 to Ubuntu 16.04.
>> >
>> > Also, this issue has been around for months. It was causing a handful
of
>> > shutdowns a week with the occasional spat of more frequent shutdowns.
It
>> > is
>> > entirely possible that the errors are behaviour related and nothing
to
>> do
>> > with the upgrade and that current email behaviour is triggering it
big
>> > style.
>> >
>> > I'm not convinced though, unfortunately I'm not convinced of anything
>> else
>> > hence not having much to go on. I can't see any consistencies in the
>> > behaviour leading up to the events.
>> >
>> > I don't think it is database related. It happened on one of the mail
>> > servers during the upgrade to 16.04 when ASSP had just started but I
had

>> > not yet got into the web interface to replace the corrupted encrypted
>> > strings with the correct values so all database connections were in
>> error.
>> >
>> > The problem has already started again this morning so I can see this
>> being
>> > another fun day that either leads to a fix or having to put something
>> else
>> > in place.
>> >
>> > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
>> >
>> > > I've been reading here, but I haven't had anything to suggest.  All
>> > seems
>> > > quite odd if it was working prior to upgrading and downgrading
didn't

>> > work.
>> > >
>> > >
>> > > Could you spin up a backup of the installation after copying the
>> current
>> > > data?  Sure you'd have an older corpus, but I'd think you could add
>> the
>> > new
>> > > files if necessary, manually replace whitelist etc.
>> > >
>> > >
>> > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
>> > >
>> > > > Further development on this today, very little.
>> > > > I have moved both servers onto Ubuntu 16.04 LTS which means going
>> from
>> > > perl
>> > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
>> > > >
>> > > > The admin user db did not work after the upgrade so I had to
empty
>> the
>> > > > tables before it would come back online.
>> > > >
>> > > > I'm still getting delayed emails and assp shutting down telling
me
>> it
>> > is
>> > > > unable
>> > > > to detect any running worker.
>> > > >
>> > > > If this goes on much longer the MD will pull the plug and we'll
end
>> up
>> > > > moving to a third party solution which is not something I want
but

>> if
>> > I
>> > > > can't fix it I can't defend it :/
>> > > >
>> > > > ------------------------------------------------------------
>> > > > ------------------
>> > > > Check out the vibrant tech community on one of the world's most
>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > _______________________________________________
>> > > > Assp-test mailing list
>> > > > [hidden email]
>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > >
>> > > >
>> > >
>> > > ------------------------------------------------------------
>> > > ------------------
>> > > Check out the vibrant tech community on one of the world's most
>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > _______________________________________________
>> > > Assp-test mailing list
>> > > [hidden email]
>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > >
>> > >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> >
>> >
>> > DISCLAIMER:
>> > *******************************************************
>> > This email and any files transmitted with it may be confidential,
>> legally
>> > privileged and protected in law and are intended solely for the use
of

>> the
>> >
>> > individual to whom it is addressed.
>> > This email was multiple times scanned for viruses. There should be no
>> > known virus in this email!
>> > *******************************************************
>> >
>> >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential,
legally
>> privileged and protected in law and are intended solely for the use of
the

>>
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
In reply to this post by Thomas Eckardt/eck
Thanks.

I've had both servers come up against unable to detect any running worker
since clearing out all the files suggested. So I'm getting 16279 running
now.

I noticed the startup with those files removed was really quick, starting
back up the second time took several minutes so presumably reading those
files during startup takes a little while versus creating them fresh.

I've caught one already and traced it through the new logs:

016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got connection request
2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up the best
Worker for new connection - 73
2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt worker Worker_3
(12) for new connection
2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread interrupted Worker_3
(12) to submit the connection
2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
for the answer of Worker_3 which handles 12 sockets
2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3: 0.0462169647216797
2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection from
MainThread - 73/73
2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_3 in 2.972 seconds - got (ok)
2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016 11:47:41
+0100
2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
0.0296478271484375
2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_3 in 31.940 seconds - got (ok)

In this case, it looks like the connection ended without any actual data.
There is lots of activity reported by Worker_3 in between 11:47:43 and
11:48:13 but this all pertains to other connections that were already in
progress.

That is a bit different to the earlier one where a message was received and
the message completed within the 30s window.

I'm not seeing anything to help me figure out why though and I don't want
to simply post a big excerpt of the maillog.txt.

On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt <[hidden email]>
wrote:

> >I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
> it
> hasn't made its way live yet.
>
> Sorry, my background CVS sync was not running - update is done.
>
> Thomas
>
>
>
>
> Von:    cw <[hidden email]>
> An:     ASSP development mailing list <[hidden email]>
> Datum:  05.10.2016 11:48
> Betreff:        Re: [Assp-test] unable to detect any running worker
>
>
>
> Thank you Thomas.
>
> useDB4IntCache - already set to off
> I've set WorkerLog to diagnostic and done the other steps.
> I don't have anything in CorrectASSPcfg.pm so as part of the
> troubleshooting I have previously deleted it and downloaded a fresh copy
> from SourceForge.
>
> I have to say that ASSP started up within seconds after clearing those
> files out - it normally takes several minutes and has always done.
>
> I looked at SF but only see 16275 in test (updated 3 days ago) so maybe it
> hasn't made its way live yet.
>
> On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
> <[hidden email]>
> wrote:
>
> > I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
> version
> > shows some more information, if 'WorkerLog' is set to diagnostic.
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:    cw <[hidden email]>
> > An:     ASSP development mailing list <[hidden email]>
> > Datum:  05.10.2016 10:10
> > Betreff:        Re: [Assp-test] unable to detect any running worker
> >
> >
> >
> > Hi Thomas,
> >
> > Thanks for chipping in. All modules are installed by running the latest
> > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
> requires
> > a minor edit to the code to get it to build. So that module was
> installed
> > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
> > make clean
> > perl Makefile.PL
> > make
> > make test
> > make install
> >
> > I run cpan-outdated -p|cpanm from time to time in order to keep modules
> up
> > to date as well so things don't stay stuck on old versions.
> >
> > Everything that is installed now is a completely fresh build as the
> > Upgrade
> > to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had
> to
> > be installed from scratch.
> >
> > I bypassed the issue by truncating the tables and setting up the users
> > again. With there only being one user it was easier to do that than muck
> > about with it - especially with the other bigger issue at hand.
> >
> > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
> > <[hidden email]>
> > wrote:
> >
> > > >to replace the corrupted encrypted
> > > strings with the correct values
> > >
> > > Was the 'Crypt::GOST' module from the SF download page at the old assp
> > > instance?
> > > If it was, did you install the 'Crypt::GOST' module from the SF
> download
> > > page, before you started the new assp instance?
> > >
> > > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
> > > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
> > >
> > > Thomas
> > >
> > >
> > >
> > >
> > >
> > > Von:    cw <[hidden email]>
> > > An:     ASSP development mailing list
> <[hidden email]>
> > > Datum:  05.10.2016 09:52
> > > Betreff:        Re: [Assp-test] unable to detect any running worker
> > >
> > >
> > >
> > > Cheers for the reply.
> > >
> > > I don't think backups are an option seen as I've moved completely from
> > > Ubuntu 14.04 to Ubuntu 16.04.
> > >
> > > Also, this issue has been around for months. It was causing a handful
> of
> > > shutdowns a week with the occasional spat of more frequent shutdowns.
> It
> > > is
> > > entirely possible that the errors are behaviour related and nothing to
> > do
> > > with the upgrade and that current email behaviour is triggering it big
> > > style.
> > >
> > > I'm not convinced though, unfortunately I'm not convinced of anything
> > else
> > > hence not having much to go on. I can't see any consistencies in the
> > > behaviour leading up to the events.
> > >
> > > I don't think it is database related. It happened on one of the mail
> > > servers during the upgrade to 16.04 when ASSP had just started but I
> had
> > > not yet got into the web interface to replace the corrupted encrypted
> > > strings with the correct values so all database connections were in
> > error.
> > >
> > > The problem has already started again this morning so I can see this
> > being
> > > another fun day that either leads to a fix or having to put something
> > else
> > > in place.
> > >
> > > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
> > >
> > > > I've been reading here, but I haven't had anything to suggest.  All
> > > seems
> > > > quite odd if it was working prior to upgrading and downgrading
> didn't
> > > work.
> > > >
> > > >
> > > > Could you spin up a backup of the installation after copying the
> > current
> > > > data?  Sure you'd have an older corpus, but I'd think you could add
> > the
> > > new
> > > > files if necessary, manually replace whitelist etc.
> > > >
> > > >
> > > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
> > > >
> > > > > Further development on this today, very little.
> > > > > I have moved both servers onto Ubuntu 16.04 LTS which means going
> > from
> > > > perl
> > > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> > > > >
> > > > > The admin user db did not work after the upgrade so I had to empty
> > the
> > > > > tables before it would come back online.
> > > > >
> > > > > I'm still getting delayed emails and assp shutting down telling me
> > it
> > > is
> > > > > unable
> > > > > to detect any running worker.
> > > > >
> > > > > If this goes on much longer the MD will pull the plug and we'll
> end
> > up
> > > > > moving to a third party solution which is not something I want but
> > if
> > > I
> > > > > can't fix it I can't defend it :/
> > > > >
> > > > > ------------------------------------------------------------
> > > > > ------------------
> > > > > Check out the vibrant tech community on one of the world's most
> > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > > _______________________________________________
> > > > > Assp-test mailing list
> > > > > [hidden email]
> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > > >
> > > > >
> > > >
> > > > ------------------------------------------------------------
> > > > ------------------
> > > > Check out the vibrant tech community on one of the world's most
> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > _______________________________________________
> > > > Assp-test mailing list
> > > > [hidden email]
> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >
> > > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > >
> > >
> > > DISCLAIMER:
> > > *******************************************************
> > > This email and any files transmitted with it may be confidential,
> > legally
> > > privileged and protected in law and are intended solely for the use of
> > the
> > >
> > > individual to whom it is addressed.
> > > This email was multiple times scanned for viruses. There should be no
> > > known virus in this email!
> > > *******************************************************
> > >
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > *******************************************************
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > *******************************************************
> >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
Hi Thomas,

The init scripts and start/stop scripts must be hanging around from an
earlier version. The start script pretty much calls /usr/bin/perl
/usr/local/assp/assp.pl /usr/local/assp

I don't think the stop script is ever actually used - there is no reference
to it anywhere in the config and my external monitoring just calls the
start script. I'll definitely look at updating them.

ASSP shuts itself down in these circumstances as follows:

2016-10-05 12:27:57 [Main_Thread] Error: Main_Thread is unable to transfer
connection to any worker within 120 seconds - restart ASSP!
2016-10-05 12:27:57 [Main_Thread] Initializing shutdown sequence
2016-10-05 12:27:57 [Shutdown] Info: removing all SMTP and Proxy listeners
2016-10-05 12:27:57 [Shutdown] Tell Worker 6 - QUIT
2016-10-05 12:27:57 [Shutdown] Tell Worker 3 - QUIT
2016-10-05 12:27:57 [Shutdown] Tell Worker 4 - QUIT
2016-10-05 12:27:57 [Shutdown] Tell Worker 5 - QUIT
2016-10-05 12:27:57 [Worker_4] Worker_4 has active connections. Will wait
until all connections are finished but max 45 seconds!
2016-10-05 12:27:57 [Worker_3] Worker_3 has active connections. Will wait
until all connections are finished but max 45 seconds!
2016-10-05 12:27:57 [Shutdown] Tell Worker 7 - QUIT
2016-10-05 12:27:57 [Shutdown] Tell Worker 1 - QUIT
2016-10-05 12:27:57 [Worker_1] Worker_1 has active connections. Will wait
until all connections are finished but max 45 seconds!
2016-10-05 12:27:57 [Worker_7] Worker_7 has active connections. Will wait
until all connections are finished but max 45 seconds!
2016-10-05 12:27:57 [Shutdown] Tell Worker 2 - QUIT
2016-10-05 12:28:47 [Shutdown] Error: at least one of the SMTP workers has
not finished work within 50 seconds
2016-10-05 12:28:47 [Shutdown] Closing all databases
2016-10-05 12:28:47 [Shutdown] Info: removing all WEB listeners
2016-10-05 12:28:47 [Shutdown] Info: shutdown reason was: restarting
2016-10-05 12:28:47 [Shutdown] ASSP finished work

So no unclean kill or anything like that. I can't remember the last time
that ASSP locked up completely and had to be forcibly terminated.

On Wed, Oct 5, 2016 at 12:14 PM, cw <[hidden email]> wrote:

> Thanks.
>
> I've had both servers come up against unable to detect any running worker
> since clearing out all the files suggested. So I'm getting 16279 running
> now.
>
> I noticed the startup with those files removed was really quick, starting
> back up the second time took several minutes so presumably reading those
> files during startup takes a little while versus creating them fresh.
>
> I've caught one already and traced it through the new logs:
>
> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got connection request
> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up the best
> Worker for new connection - 73
> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt worker Worker_3
> (12) for new connection
> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread interrupted Worker_3
> (12) to submit the connection
> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
> for the answer of Worker_3 which handles 12 sockets
> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3: 0.0462169647216797
> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection from
> MainThread - 73/73
> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
> Worker_3 in 2.972 seconds - got (ok)
> 2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016 11:47:41
> +0100
> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
> 0.0296478271484375
> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
> Worker_3 in 31.940 seconds - got (ok)
>
> In this case, it looks like the connection ended without any actual data.
> There is lots of activity reported by Worker_3 in between 11:47:43 and
> 11:48:13 but this all pertains to other connections that were already in
> progress.
>
> That is a bit different to the earlier one where a message was received
> and the message completed within the 30s window.
>
> I'm not seeing anything to help me figure out why though and I don't want
> to simply post a big excerpt of the maillog.txt.
>
> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt <
> [hidden email]> wrote:
>
>> >I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
>> it
>> hasn't made its way live yet.
>>
>> Sorry, my background CVS sync was not running - update is done.
>>
>> Thomas
>>
>>
>>
>>
>> Von:    cw <[hidden email]>
>> An:     ASSP development mailing list <[hidden email]>
>> Datum:  05.10.2016 11:48
>> Betreff:        Re: [Assp-test] unable to detect any running worker
>>
>>
>>
>> Thank you Thomas.
>>
>> useDB4IntCache - already set to off
>> I've set WorkerLog to diagnostic and done the other steps.
>> I don't have anything in CorrectASSPcfg.pm so as part of the
>> troubleshooting I have previously deleted it and downloaded a fresh copy
>> from SourceForge.
>>
>> I have to say that ASSP started up within seconds after clearing those
>> files out - it normally takes several minutes and has always done.
>>
>> I looked at SF but only see 16275 in test (updated 3 days ago) so maybe it
>> hasn't made its way live yet.
>>
>> On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
>> <[hidden email]>
>> wrote:
>>
>> > I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
>> version
>> > shows some more information, if 'WorkerLog' is set to diagnostic.
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> >
>> > Von:    cw <[hidden email]>
>> > An:     ASSP development mailing list <[hidden email]>
>> > Datum:  05.10.2016 10:10
>> > Betreff:        Re: [Assp-test] unable to detect any running worker
>> >
>> >
>> >
>> > Hi Thomas,
>> >
>> > Thanks for chipping in. All modules are installed by running the latest
>> > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
>> requires
>> > a minor edit to the code to get it to build. So that module was
>> installed
>> > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
>> > make clean
>> > perl Makefile.PL
>> > make
>> > make test
>> > make install
>> >
>> > I run cpan-outdated -p|cpanm from time to time in order to keep modules
>> up
>> > to date as well so things don't stay stuck on old versions.
>> >
>> > Everything that is installed now is a completely fresh build as the
>> > Upgrade
>> > to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had
>> to
>> > be installed from scratch.
>> >
>> > I bypassed the issue by truncating the tables and setting up the users
>> > again. With there only being one user it was easier to do that than muck
>> > about with it - especially with the other bigger issue at hand.
>> >
>> > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
>> > <[hidden email]>
>> > wrote:
>> >
>> > > >to replace the corrupted encrypted
>> > > strings with the correct values
>> > >
>> > > Was the 'Crypt::GOST' module from the SF download page at the old assp
>> > > instance?
>> > > If it was, did you install the 'Crypt::GOST' module from the SF
>> download
>> > > page, before you started the new assp instance?
>> > >
>> > > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
>> > > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>> > >
>> > > Thomas
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Von:    cw <[hidden email]>
>> > > An:     ASSP development mailing list
>> <[hidden email]>
>> > > Datum:  05.10.2016 09:52
>> > > Betreff:        Re: [Assp-test] unable to detect any running worker
>> > >
>> > >
>> > >
>> > > Cheers for the reply.
>> > >
>> > > I don't think backups are an option seen as I've moved completely from
>> > > Ubuntu 14.04 to Ubuntu 16.04.
>> > >
>> > > Also, this issue has been around for months. It was causing a handful
>> of
>> > > shutdowns a week with the occasional spat of more frequent shutdowns.
>> It
>> > > is
>> > > entirely possible that the errors are behaviour related and nothing to
>> > do
>> > > with the upgrade and that current email behaviour is triggering it big
>> > > style.
>> > >
>> > > I'm not convinced though, unfortunately I'm not convinced of anything
>> > else
>> > > hence not having much to go on. I can't see any consistencies in the
>> > > behaviour leading up to the events.
>> > >
>> > > I don't think it is database related. It happened on one of the mail
>> > > servers during the upgrade to 16.04 when ASSP had just started but I
>> had
>> > > not yet got into the web interface to replace the corrupted encrypted
>> > > strings with the correct values so all database connections were in
>> > error.
>> > >
>> > > The problem has already started again this morning so I can see this
>> > being
>> > > another fun day that either leads to a fix or having to put something
>> > else
>> > > in place.
>> > >
>> > > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
>> > >
>> > > > I've been reading here, but I haven't had anything to suggest.  All
>> > > seems
>> > > > quite odd if it was working prior to upgrading and downgrading
>> didn't
>> > > work.
>> > > >
>> > > >
>> > > > Could you spin up a backup of the installation after copying the
>> > current
>> > > > data?  Sure you'd have an older corpus, but I'd think you could add
>> > the
>> > > new
>> > > > files if necessary, manually replace whitelist etc.
>> > > >
>> > > >
>> > > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
>> > > >
>> > > > > Further development on this today, very little.
>> > > > > I have moved both servers onto Ubuntu 16.04 LTS which means going
>> > from
>> > > > perl
>> > > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
>> > > > >
>> > > > > The admin user db did not work after the upgrade so I had to empty
>> > the
>> > > > > tables before it would come back online.
>> > > > >
>> > > > > I'm still getting delayed emails and assp shutting down telling me
>> > it
>> > > is
>> > > > > unable
>> > > > > to detect any running worker.
>> > > > >
>> > > > > If this goes on much longer the MD will pull the plug and we'll
>> end
>> > up
>> > > > > moving to a third party solution which is not something I want but
>> > if
>> > > I
>> > > > > can't fix it I can't defend it :/
>> > > > >
>> > > > > ------------------------------------------------------------
>> > > > > ------------------
>> > > > > Check out the vibrant tech community on one of the world's most
>> > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > > _______________________________________________
>> > > > > Assp-test mailing list
>> > > > > [hidden email]
>> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > > >
>> > > > >
>> > > >
>> > > > ------------------------------------------------------------
>> > > > ------------------
>> > > > Check out the vibrant tech community on one of the world's most
>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > _______________________________________________
>> > > > Assp-test mailing list
>> > > > [hidden email]
>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > >
>> > > >
>> > > ------------------------------------------------------------
>> > > ------------------
>> > > Check out the vibrant tech community on one of the world's most
>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > _______________________________________________
>> > > Assp-test mailing list
>> > > [hidden email]
>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > >
>> > >
>> > >
>> > >
>> > > DISCLAIMER:
>> > > *******************************************************
>> > > This email and any files transmitted with it may be confidential,
>> > legally
>> > > privileged and protected in law and are intended solely for the use of
>> > the
>> > >
>> > > individual to whom it is addressed.
>> > > This email was multiple times scanned for viruses. There should be no
>> > > known virus in this email!
>> > > *******************************************************
>> > >
>> > >
>> > > ------------------------------------------------------------
>> > > ------------------
>> > > Check out the vibrant tech community on one of the world's most
>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > _______________________________________________
>> > > Assp-test mailing list
>> > > [hidden email]
>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > >
>> > >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> >
>> >
>> > DISCLAIMER:
>> > *******************************************************
>> > This email and any files transmitted with it may be confidential,
>> legally
>> > privileged and protected in law and are intended solely for the use of
>> the
>> >
>> > individual to whom it is addressed.
>> > This email was multiple times scanned for viruses. There should be no
>> > known virus in this email!
>> > *******************************************************
>> >
>> >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential, legally
>> privileged and protected in law and are intended solely for the use of the
>>
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
Hi Thomas.

The start and stop scripts are actually in the rc folder called assp.dat
start.dat and stop.dat.

They must have been current when I first set things up I've moved
everything over to the Debian script and it didn't take long for both ASSP
instances to do exactly the same as before so these have nothing to do with
it I'm afraid.

All the best,
Colin.




On Wed, Oct 5, 2016 at 12:38 PM, cw <[hidden email]> wrote:

> Hi Thomas,
>
> The init scripts and start/stop scripts must be hanging around from an
> earlier version. The start script pretty much calls /usr/bin/perl
> /usr/local/assp/assp.pl /usr/local/assp
>
> I don't think the stop script is ever actually used - there is no
> reference to it anywhere in the config and my external monitoring just
> calls the start script. I'll definitely look at updating them.
>
> ASSP shuts itself down in these circumstances as follows:
>
> 2016-10-05 12:27:57 [Main_Thread] Error: Main_Thread is unable to transfer
> connection to any worker within 120 seconds - restart ASSP!
> 2016-10-05 12:27:57 [Main_Thread] Initializing shutdown sequence
> 2016-10-05 12:27:57 [Shutdown] Info: removing all SMTP and Proxy listeners
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 6 - QUIT
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 3 - QUIT
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 4 - QUIT
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 5 - QUIT
> 2016-10-05 12:27:57 [Worker_4] Worker_4 has active connections. Will wait
> until all connections are finished but max 45 seconds!
> 2016-10-05 12:27:57 [Worker_3] Worker_3 has active connections. Will wait
> until all connections are finished but max 45 seconds!
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 7 - QUIT
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 1 - QUIT
> 2016-10-05 12:27:57 [Worker_1] Worker_1 has active connections. Will wait
> until all connections are finished but max 45 seconds!
> 2016-10-05 12:27:57 [Worker_7] Worker_7 has active connections. Will wait
> until all connections are finished but max 45 seconds!
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 2 - QUIT
> 2016-10-05 12:28:47 [Shutdown] Error: at least one of the SMTP workers has
> not finished work within 50 seconds
> 2016-10-05 12:28:47 [Shutdown] Closing all databases
> 2016-10-05 12:28:47 [Shutdown] Info: removing all WEB listeners
> 2016-10-05 12:28:47 [Shutdown] Info: shutdown reason was: restarting
> 2016-10-05 12:28:47 [Shutdown] ASSP finished work
>
> So no unclean kill or anything like that. I can't remember the last time
> that ASSP locked up completely and had to be forcibly terminated.
>
> On Wed, Oct 5, 2016 at 12:14 PM, cw <[hidden email]> wrote:
>
>> Thanks.
>>
>> I've had both servers come up against unable to detect any running worker
>> since clearing out all the files suggested. So I'm getting 16279 running
>> now.
>>
>> I noticed the startup with those files removed was really quick, starting
>> back up the second time took several minutes so presumably reading those
>> files during startup takes a little while versus creating them fresh.
>>
>> I've caught one already and traced it through the new logs:
>>
>> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got connection request
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up the best
>> Worker for new connection - 73
>> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt worker Worker_3
>> (12) for new connection
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread interrupted Worker_3
>> (12) to submit the connection
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
>> for the answer of Worker_3 which handles 12 sockets
>> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3: 0.0462169647216797
>> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection from
>> MainThread - 73/73
>> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
>> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
>> Worker_3 in 2.972 seconds - got (ok)
>> 2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
>> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
>> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016 11:47:41
>> +0100
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
>> 0.0296478271484375
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
>> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
>> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
>> Worker_3 in 31.940 seconds - got (ok)
>>
>> In this case, it looks like the connection ended without any actual data.
>> There is lots of activity reported by Worker_3 in between 11:47:43 and
>> 11:48:13 but this all pertains to other connections that were already in
>> progress.
>>
>> That is a bit different to the earlier one where a message was received
>> and the message completed within the 30s window.
>>
>> I'm not seeing anything to help me figure out why though and I don't want
>> to simply post a big excerpt of the maillog.txt.
>>
>> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt <
>> [hidden email]> wrote:
>>
>>> >I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
>>> it
>>> hasn't made its way live yet.
>>>
>>> Sorry, my background CVS sync was not running - update is done.
>>>
>>> Thomas
>>>
>>>
>>>
>>>
>>> Von:    cw <[hidden email]>
>>> An:     ASSP development mailing list <[hidden email]>
>>> Datum:  05.10.2016 11:48
>>> Betreff:        Re: [Assp-test] unable to detect any running worker
>>>
>>>
>>>
>>> Thank you Thomas.
>>>
>>> useDB4IntCache - already set to off
>>> I've set WorkerLog to diagnostic and done the other steps.
>>> I don't have anything in CorrectASSPcfg.pm so as part of the
>>> troubleshooting I have previously deleted it and downloaded a fresh copy
>>> from SourceForge.
>>>
>>> I have to say that ASSP started up within seconds after clearing those
>>> files out - it normally takes several minutes and has always done.
>>>
>>> I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
>>> it
>>> hasn't made its way live yet.
>>>
>>> On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
>>> <[hidden email]>
>>> wrote:
>>>
>>> > I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
>>> version
>>> > shows some more information, if 'WorkerLog' is set to diagnostic.
>>> >
>>> > Thomas
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Von:    cw <[hidden email]>
>>> > An:     ASSP development mailing list <[hidden email]
>>> >
>>> > Datum:  05.10.2016 10:10
>>> > Betreff:        Re: [Assp-test] unable to detect any running worker
>>> >
>>> >
>>> >
>>> > Hi Thomas,
>>> >
>>> > Thanks for chipping in. All modules are installed by running the latest
>>> > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
>>> requires
>>> > a minor edit to the code to get it to build. So that module was
>>> installed
>>> > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
>>> > make clean
>>> > perl Makefile.PL
>>> > make
>>> > make test
>>> > make install
>>> >
>>> > I run cpan-outdated -p|cpanm from time to time in order to keep modules
>>> up
>>> > to date as well so things don't stay stuck on old versions.
>>> >
>>> > Everything that is installed now is a completely fresh build as the
>>> > Upgrade
>>> > to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had
>>> to
>>> > be installed from scratch.
>>> >
>>> > I bypassed the issue by truncating the tables and setting up the users
>>> > again. With there only being one user it was easier to do that than
>>> muck
>>> > about with it - especially with the other bigger issue at hand.
>>> >
>>> > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
>>> > <[hidden email]>
>>> > wrote:
>>> >
>>> > > >to replace the corrupted encrypted
>>> > > strings with the correct values
>>> > >
>>> > > Was the 'Crypt::GOST' module from the SF download page at the old
>>> assp
>>> > > instance?
>>> > > If it was, did you install the 'Crypt::GOST' module from the SF
>>> download
>>> > > page, before you started the new assp instance?
>>> > >
>>> > > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
>>> > > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>>> > >
>>> > > Thomas
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Von:    cw <[hidden email]>
>>> > > An:     ASSP development mailing list
>>> <[hidden email]>
>>> > > Datum:  05.10.2016 09:52
>>> > > Betreff:        Re: [Assp-test] unable to detect any running worker
>>> > >
>>> > >
>>> > >
>>> > > Cheers for the reply.
>>> > >
>>> > > I don't think backups are an option seen as I've moved completely
>>> from
>>> > > Ubuntu 14.04 to Ubuntu 16.04.
>>> > >
>>> > > Also, this issue has been around for months. It was causing a handful
>>> of
>>> > > shutdowns a week with the occasional spat of more frequent shutdowns.
>>> It
>>> > > is
>>> > > entirely possible that the errors are behaviour related and nothing
>>> to
>>> > do
>>> > > with the upgrade and that current email behaviour is triggering it
>>> big
>>> > > style.
>>> > >
>>> > > I'm not convinced though, unfortunately I'm not convinced of anything
>>> > else
>>> > > hence not having much to go on. I can't see any consistencies in the
>>> > > behaviour leading up to the events.
>>> > >
>>> > > I don't think it is database related. It happened on one of the mail
>>> > > servers during the upgrade to 16.04 when ASSP had just started but I
>>> had
>>> > > not yet got into the web interface to replace the corrupted encrypted
>>> > > strings with the correct values so all database connections were in
>>> > error.
>>> > >
>>> > > The problem has already started again this morning so I can see this
>>> > being
>>> > > another fun day that either leads to a fix or having to put something
>>> > else
>>> > > in place.
>>> > >
>>> > > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
>>> > >
>>> > > > I've been reading here, but I haven't had anything to suggest.  All
>>> > > seems
>>> > > > quite odd if it was working prior to upgrading and downgrading
>>> didn't
>>> > > work.
>>> > > >
>>> > > >
>>> > > > Could you spin up a backup of the installation after copying the
>>> > current
>>> > > > data?  Sure you'd have an older corpus, but I'd think you could add
>>> > the
>>> > > new
>>> > > > files if necessary, manually replace whitelist etc.
>>> > > >
>>> > > >
>>> > > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
>>> > > >
>>> > > > > Further development on this today, very little.
>>> > > > > I have moved both servers onto Ubuntu 16.04 LTS which means going
>>> > from
>>> > > > perl
>>> > > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
>>> > > > >
>>> > > > > The admin user db did not work after the upgrade so I had to
>>> empty
>>> > the
>>> > > > > tables before it would come back online.
>>> > > > >
>>> > > > > I'm still getting delayed emails and assp shutting down telling
>>> me
>>> > it
>>> > > is
>>> > > > > unable
>>> > > > > to detect any running worker.
>>> > > > >
>>> > > > > If this goes on much longer the MD will pull the plug and we'll
>>> end
>>> > up
>>> > > > > moving to a third party solution which is not something I want
>>> but
>>> > if
>>> > > I
>>> > > > > can't fix it I can't defend it :/
>>> > > > >
>>> > > > > ------------------------------------------------------------
>>> > > > > ------------------
>>> > > > > Check out the vibrant tech community on one of the world's most
>>> > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > > > > _______________________________________________
>>> > > > > Assp-test mailing list
>>> > > > > [hidden email]
>>> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> > > > >
>>> > > > >
>>> > > >
>>> > > > ------------------------------------------------------------
>>> > > > ------------------
>>> > > > Check out the vibrant tech community on one of the world's most
>>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > > > _______________________________________________
>>> > > > Assp-test mailing list
>>> > > > [hidden email]
>>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> > > >
>>> > > >
>>> > > ------------------------------------------------------------
>>> > > ------------------
>>> > > Check out the vibrant tech community on one of the world's most
>>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > > _______________________________________________
>>> > > Assp-test mailing list
>>> > > [hidden email]
>>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > DISCLAIMER:
>>> > > *******************************************************
>>> > > This email and any files transmitted with it may be confidential,
>>> > legally
>>> > > privileged and protected in law and are intended solely for the use
>>> of
>>> > the
>>> > >
>>> > > individual to whom it is addressed.
>>> > > This email was multiple times scanned for viruses. There should be no
>>> > > known virus in this email!
>>> > > *******************************************************
>>> > >
>>> > >
>>> > > ------------------------------------------------------------
>>> > > ------------------
>>> > > Check out the vibrant tech community on one of the world's most
>>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > > _______________________________________________
>>> > > Assp-test mailing list
>>> > > [hidden email]
>>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> > >
>>> > >
>>> > ------------------------------------------------------------
>>> > ------------------
>>> > Check out the vibrant tech community on one of the world's most
>>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > _______________________________________________
>>> > Assp-test mailing list
>>> > [hidden email]
>>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> >
>>> >
>>> >
>>> >
>>> > DISCLAIMER:
>>> > *******************************************************
>>> > This email and any files transmitted with it may be confidential,
>>> legally
>>> > privileged and protected in law and are intended solely for the use of
>>> the
>>> >
>>> > individual to whom it is addressed.
>>> > This email was multiple times scanned for viruses. There should be no
>>> > known virus in this email!
>>> > *******************************************************
>>> >
>>> >
>>> > ------------------------------------------------------------
>>> > ------------------
>>> > Check out the vibrant tech community on one of the world's most
>>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > _______________________________________________
>>> > Assp-test mailing list
>>> > [hidden email]
>>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> >
>>> >
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Assp-test mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>>
>>>
>>>
>>>
>>> DISCLAIMER:
>>> *******************************************************
>>> This email and any files transmitted with it may be confidential, legally
>>> privileged and protected in law and are intended solely for the use of
>>> the
>>>
>>> individual to whom it is addressed.
>>> This email was multiple times scanned for viruses. There should be no
>>> known virus in this email!
>>> *******************************************************
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Assp-test mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>>
>>>
>>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Thomas Eckardt/eck
In reply to this post by Colin
It seems, that this system is overloaded with active connections.

>2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_3 in 2.972 seconds - got (ok)

Even this time value is ten times higher than expected.

>2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170

The '170' is the filehandle number in perl (counted from 1.....) - this
implies 70 or more active connections.

>2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
for the answer of Worker_3 which handles 12 sockets

This implies the same, if the equal balancing of all connections over all
seven workers works correctly.

>2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
>2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
>2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
>2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648

The processing time itself seems to be normal. SC-Time - is the time
required to process one loop (read + queue or read + check + queue) for
one active connection.

Assuming an average of 4 second life time per connection - the workload
per day would be 1.5 million (70 / 4 *3600 *24) connections based on the
provided output.
Configured to an ISP mode the avg. maximum for a single assp instance is
800.000 connections. To be able to handle workload peeks, the
configuration should not allow more than 400.000 connection a day.

ISP mode:
- 16 CPU cores
- 16 Workers
- 24 GB RAM
- dedicated systems for MTA, enterprise database, ClamAV
- HMM and spamDB holded unshared in RAM
- high performance DNS servers
- limited DNS usage (disable most extensive DNS-using checks - eg. URIBL)
- no OCR
- no DCC
- no Razor2

>2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_3 in 31.940 seconds - got (ok)

This time value is near the end of the line. At this time a small count of
additionally connections can lead in to an assp shutdown, because the
workers don't accept new connections.

Thomas



Von:    cw <[hidden email]>
An:     ASSP development mailing list <[hidden email]>
Datum:  05.10.2016 13:15
Betreff:        Re: [Assp-test] unable to detect any running worker



Thanks.

I've had both servers come up against unable to detect any running worker
since clearing out all the files suggested. So I'm getting 16279 running
now.

I noticed the startup with those files removed was really quick, starting
back up the second time took several minutes so presumably reading those
files during startup takes a little while versus creating them fresh.

I've caught one already and traced it through the new logs:

016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got connection request
2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up the best
Worker for new connection - 73
2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt worker Worker_3
(12) for new connection
2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread interrupted Worker_3
(12) to submit the connection
2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
for the answer of Worker_3 which handles 12 sockets
2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3: 0.0462169647216797
2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection from
MainThread - 73/73
2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_3 in 2.972 seconds - got (ok)
2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016 11:47:41
+0100
2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
0.0296478271484375
2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_3 in 31.940 seconds - got (ok)

In this case, it looks like the connection ended without any actual data.
There is lots of activity reported by Worker_3 in between 11:47:43 and
11:48:13 but this all pertains to other connections that were already in
progress.

That is a bit different to the earlier one where a message was received
and
the message completed within the 30s window.

I'm not seeing anything to help me figure out why though and I don't want
to simply post a big excerpt of the maillog.txt.

On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt
<[hidden email]>
wrote:

> >I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
> it
> hasn't made its way live yet.
>
> Sorry, my background CVS sync was not running - update is done.
>
> Thomas
>
>
>
>
> Von:    cw <[hidden email]>
> An:     ASSP development mailing list <[hidden email]>
> Datum:  05.10.2016 11:48
> Betreff:        Re: [Assp-test] unable to detect any running worker
>
>
>
> Thank you Thomas.
>
> useDB4IntCache - already set to off
> I've set WorkerLog to diagnostic and done the other steps.
> I don't have anything in CorrectASSPcfg.pm so as part of the
> troubleshooting I have previously deleted it and downloaded a fresh copy
> from SourceForge.
>
> I have to say that ASSP started up within seconds after clearing those
> files out - it normally takes several minutes and has always done.
>
> I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
it

> hasn't made its way live yet.
>
> On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
> <[hidden email]>
> wrote:
>
> > I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
> version
> > shows some more information, if 'WorkerLog' is set to diagnostic.
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:    cw <[hidden email]>
> > An:     ASSP development mailing list
<[hidden email]>
> > Datum:  05.10.2016 10:10
> > Betreff:        Re: [Assp-test] unable to detect any running worker
> >
> >
> >
> > Hi Thomas,
> >
> > Thanks for chipping in. All modules are installed by running the
latest

> > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
> requires
> > a minor edit to the code to get it to build. So that module was
> installed
> > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
> > make clean
> > perl Makefile.PL
> > make
> > make test
> > make install
> >
> > I run cpan-outdated -p|cpanm from time to time in order to keep
modules
> up
> > to date as well so things don't stay stuck on old versions.
> >
> > Everything that is installed now is a completely fresh build as the
> > Upgrade
> > to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore
had
> to
> > be installed from scratch.
> >
> > I bypassed the issue by truncating the tables and setting up the users
> > again. With there only being one user it was easier to do that than
muck

> > about with it - especially with the other bigger issue at hand.
> >
> > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
> > <[hidden email]>
> > wrote:
> >
> > > >to replace the corrupted encrypted
> > > strings with the correct values
> > >
> > > Was the 'Crypt::GOST' module from the SF download page at the old
assp

> > > instance?
> > > If it was, did you install the 'Crypt::GOST' module from the SF
> download
> > > page, before you started the new assp instance?
> > >
> > > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
> > > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
> > >
> > > Thomas
> > >
> > >
> > >
> > >
> > >
> > > Von:    cw <[hidden email]>
> > > An:     ASSP development mailing list
> <[hidden email]>
> > > Datum:  05.10.2016 09:52
> > > Betreff:        Re: [Assp-test] unable to detect any running worker
> > >
> > >
> > >
> > > Cheers for the reply.
> > >
> > > I don't think backups are an option seen as I've moved completely
from
> > > Ubuntu 14.04 to Ubuntu 16.04.
> > >
> > > Also, this issue has been around for months. It was causing a
handful
> of
> > > shutdowns a week with the occasional spat of more frequent
shutdowns.
> It
> > > is
> > > entirely possible that the errors are behaviour related and nothing
to
> > do
> > > with the upgrade and that current email behaviour is triggering it
big
> > > style.
> > >
> > > I'm not convinced though, unfortunately I'm not convinced of
anything
> > else
> > > hence not having much to go on. I can't see any consistencies in the
> > > behaviour leading up to the events.
> > >
> > > I don't think it is database related. It happened on one of the mail
> > > servers during the upgrade to 16.04 when ASSP had just started but I
> had
> > > not yet got into the web interface to replace the corrupted
encrypted
> > > strings with the correct values so all database connections were in
> > error.
> > >
> > > The problem has already started again this morning so I can see this
> > being
> > > another fun day that either leads to a fix or having to put
something

> > else
> > > in place.
> > >
> > > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
> > >
> > > > I've been reading here, but I haven't had anything to suggest. All
> > > seems
> > > > quite odd if it was working prior to upgrading and downgrading
> didn't
> > > work.
> > > >
> > > >
> > > > Could you spin up a backup of the installation after copying the
> > current
> > > > data?  Sure you'd have an older corpus, but I'd think you could
add
> > the
> > > new
> > > > files if necessary, manually replace whitelist etc.
> > > >
> > > >
> > > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
> > > >
> > > > > Further development on this today, very little.
> > > > > I have moved both servers onto Ubuntu 16.04 LTS which means
going
> > from
> > > > perl
> > > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> > > > >
> > > > > The admin user db did not work after the upgrade so I had to
empty
> > the
> > > > > tables before it would come back online.
> > > > >
> > > > > I'm still getting delayed emails and assp shutting down telling
me
> > it
> > > is
> > > > > unable
> > > > > to detect any running worker.
> > > > >
> > > > > If this goes on much longer the MD will pull the plug and we'll
> end
> > up
> > > > > moving to a third party solution which is not something I want
but

> > if
> > > I
> > > > > can't fix it I can't defend it :/
> > > > >
> > > > > ------------------------------------------------------------
> > > > > ------------------
> > > > > Check out the vibrant tech community on one of the world's most
> > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > > _______________________________________________
> > > > > Assp-test mailing list
> > > > > [hidden email]
> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > > >
> > > > >
> > > >
> > > > ------------------------------------------------------------
> > > > ------------------
> > > > Check out the vibrant tech community on one of the world's most
> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > _______________________________________________
> > > > Assp-test mailing list
> > > > [hidden email]
> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >
> > > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > >
> > >
> > > DISCLAIMER:
> > > *******************************************************
> > > This email and any files transmitted with it may be confidential,
> > legally
> > > privileged and protected in law and are intended solely for the use
of
> > the
> > >
> > > individual to whom it is addressed.
> > > This email was multiple times scanned for viruses. There should be
no

> > > known virus in this email!
> > > *******************************************************
> > >
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > *******************************************************
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > *******************************************************
> >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential,
legally
> privileged and protected in law and are intended solely for the use of
the

>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
Hi Thomas,

Thank you for the rundown. The load is exceptionally high at the moment
because everything backs up every time the mail servers stop. Having said
that, I've just watched it happen again on one of the servers. It only had
7 messages showing in the shutdown_list when it happened. Looking at the
logs I see:

2016-10-05 14:23:50 [Worker_4] Info: Worker_4 got connection from
MainThread - 78/78
2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by interrupted
Worker_4 in 10.758 seconds - got (ok)
2016-10-05 14:23:50 [Worker_4] Info: Worker_4 freed Main_Thread - 101

So something is artificially inflating the connection count. My best guess
ties in with something else I was looking at where a remote server was
opening two connections that never got answered. It only got through on the
third connection and that could be replicated each time. I put it down to
some odd firewall on their end but maybe I was wrong in doing that.

The actual stats on one mailserver are:

Last reset 8 days, 0 hours, 6 mins
Messages process 165413 (20664 per day)
SMTP Connections received 113278

The other is:
Last reset 8 days, 0 hours, 18 mins
Messaged processed 131720 (16438 per day)
SMTP Connections received 93088

The stats reset themselves during on of the upgrades of ASSP, not sure why.

Each server is 16 CPU cores, 12GB RAM. DB is on a separate MySQL box though
MTA and Clam run local. URIBL is disabled. I don't use OCR, DCC or Razor. I
tried to get OCR working once and all hell broke lose and I haven't been
back since.

I've been monitoring the performance of the servers all along and not
seeing any bottlenecks in CPU, RAM or network.When the problem started the
mail servers had 8 and 10GB RAM respectively (10GB for the one running
rebuild).
All the best,
Colin



On Wed, Oct 5, 2016 at 2:07 PM, Thomas Eckardt <[hidden email]>
wrote:

> It seems, that this system is overloaded with active connections.
>
> >2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
> Worker_3 in 2.972 seconds - got (ok)
>
> Even this time value is ten times higher than expected.
>
> >2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
>
> The '170' is the filehandle number in perl (counted from 1.....) - this
> implies 70 or more active connections.
>
> >2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
> for the answer of Worker_3 which handles 12 sockets
>
> This implies the same, if the equal balancing of all connections over all
> seven workers works correctly.
>
> >2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
>
> The processing time itself seems to be normal. SC-Time - is the time
> required to process one loop (read + queue or read + check + queue) for
> one active connection.
>
> Assuming an average of 4 second life time per connection - the workload
> per day would be 1.5 million (70 / 4 *3600 *24) connections based on the
> provided output.
> Configured to an ISP mode the avg. maximum for a single assp instance is
> 800.000 connections. To be able to handle workload peeks, the
> configuration should not allow more than 400.000 connection a day.
>
> ISP mode:
> - 16 CPU cores
> - 16 Workers
> - 24 GB RAM
> - dedicated systems for MTA, enterprise database, ClamAV
> - HMM and spamDB holded unshared in RAM
> - high performance DNS servers
> - limited DNS usage (disable most extensive DNS-using checks - eg. URIBL)
> - no OCR
> - no DCC
> - no Razor2
>
> >2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
> Worker_3 in 31.940 seconds - got (ok)
>
> This time value is near the end of the line. At this time a small count of
> additionally connections can lead in to an assp shutdown, because the
> workers don't accept new connections.
>
> Thomas
>
>
>
> Von:    cw <[hidden email]>
> An:     ASSP development mailing list <[hidden email]>
> Datum:  05.10.2016 13:15
> Betreff:        Re: [Assp-test] unable to detect any running worker
>
>
>
> Thanks.
>
> I've had both servers come up against unable to detect any running worker
> since clearing out all the files suggested. So I'm getting 16279 running
> now.
>
> I noticed the startup with those files removed was really quick, starting
> back up the second time took several minutes so presumably reading those
> files during startup takes a little while versus creating them fresh.
>
> I've caught one already and traced it through the new logs:
>
> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got connection request
> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up the best
> Worker for new connection - 73
> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt worker Worker_3
> (12) for new connection
> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread interrupted Worker_3
> (12) to submit the connection
> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
> for the answer of Worker_3 which handles 12 sockets
> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3: 0.0462169647216797
> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection from
> MainThread - 73/73
> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
> Worker_3 in 2.972 seconds - got (ok)
> 2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016 11:47:41
> +0100
> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
> 0.0296478271484375
> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
> Worker_3 in 31.940 seconds - got (ok)
>
> In this case, it looks like the connection ended without any actual data.
> There is lots of activity reported by Worker_3 in between 11:47:43 and
> 11:48:13 but this all pertains to other connections that were already in
> progress.
>
> That is a bit different to the earlier one where a message was received
> and
> the message completed within the 30s window.
>
> I'm not seeing anything to help me figure out why though and I don't want
> to simply post a big excerpt of the maillog.txt.
>
> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt
> <[hidden email]>
> wrote:
>
> > >I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
> > it
> > hasn't made its way live yet.
> >
> > Sorry, my background CVS sync was not running - update is done.
> >
> > Thomas
> >
> >
> >
> >
> > Von:    cw <[hidden email]>
> > An:     ASSP development mailing list <[hidden email]>
> > Datum:  05.10.2016 11:48
> > Betreff:        Re: [Assp-test] unable to detect any running worker
> >
> >
> >
> > Thank you Thomas.
> >
> > useDB4IntCache - already set to off
> > I've set WorkerLog to diagnostic and done the other steps.
> > I don't have anything in CorrectASSPcfg.pm so as part of the
> > troubleshooting I have previously deleted it and downloaded a fresh copy
> > from SourceForge.
> >
> > I have to say that ASSP started up within seconds after clearing those
> > files out - it normally takes several minutes and has always done.
> >
> > I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
> it
> > hasn't made its way live yet.
> >
> > On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
> > <[hidden email]>
> > wrote:
> >
> > > I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
> > version
> > > shows some more information, if 'WorkerLog' is set to diagnostic.
> > >
> > > Thomas
> > >
> > >
> > >
> > >
> > >
> > > Von:    cw <[hidden email]>
> > > An:     ASSP development mailing list
> <[hidden email]>
> > > Datum:  05.10.2016 10:10
> > > Betreff:        Re: [Assp-test] unable to detect any running worker
> > >
> > >
> > >
> > > Hi Thomas,
> > >
> > > Thanks for chipping in. All modules are installed by running the
> latest
> > > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
> > requires
> > > a minor edit to the code to get it to build. So that module was
> > installed
> > > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
> > > make clean
> > > perl Makefile.PL
> > > make
> > > make test
> > > make install
> > >
> > > I run cpan-outdated -p|cpanm from time to time in order to keep
> modules
> > up
> > > to date as well so things don't stay stuck on old versions.
> > >
> > > Everything that is installed now is a completely fresh build as the
> > > Upgrade
> > > to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore
> had
> > to
> > > be installed from scratch.
> > >
> > > I bypassed the issue by truncating the tables and setting up the users
> > > again. With there only being one user it was easier to do that than
> muck
> > > about with it - especially with the other bigger issue at hand.
> > >
> > > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
> > > <[hidden email]>
> > > wrote:
> > >
> > > > >to replace the corrupted encrypted
> > > > strings with the correct values
> > > >
> > > > Was the 'Crypt::GOST' module from the SF download page at the old
> assp
> > > > instance?
> > > > If it was, did you install the 'Crypt::GOST' module from the SF
> > download
> > > > page, before you started the new assp instance?
> > > >
> > > > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
> > > > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
> > > >
> > > > Thomas
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Von:    cw <[hidden email]>
> > > > An:     ASSP development mailing list
> > <[hidden email]>
> > > > Datum:  05.10.2016 09:52
> > > > Betreff:        Re: [Assp-test] unable to detect any running worker
> > > >
> > > >
> > > >
> > > > Cheers for the reply.
> > > >
> > > > I don't think backups are an option seen as I've moved completely
> from
> > > > Ubuntu 14.04 to Ubuntu 16.04.
> > > >
> > > > Also, this issue has been around for months. It was causing a
> handful
> > of
> > > > shutdowns a week with the occasional spat of more frequent
> shutdowns.
> > It
> > > > is
> > > > entirely possible that the errors are behaviour related and nothing
> to
> > > do
> > > > with the upgrade and that current email behaviour is triggering it
> big
> > > > style.
> > > >
> > > > I'm not convinced though, unfortunately I'm not convinced of
> anything
> > > else
> > > > hence not having much to go on. I can't see any consistencies in the
> > > > behaviour leading up to the events.
> > > >
> > > > I don't think it is database related. It happened on one of the mail
> > > > servers during the upgrade to 16.04 when ASSP had just started but I
> > had
> > > > not yet got into the web interface to replace the corrupted
> encrypted
> > > > strings with the correct values so all database connections were in
> > > error.
> > > >
> > > > The problem has already started again this morning so I can see this
> > > being
> > > > another fun day that either leads to a fix or having to put
> something
> > > else
> > > > in place.
> > > >
> > > > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
> > > >
> > > > > I've been reading here, but I haven't had anything to suggest. All
> > > > seems
> > > > > quite odd if it was working prior to upgrading and downgrading
> > didn't
> > > > work.
> > > > >
> > > > >
> > > > > Could you spin up a backup of the installation after copying the
> > > current
> > > > > data?  Sure you'd have an older corpus, but I'd think you could
> add
> > > the
> > > > new
> > > > > files if necessary, manually replace whitelist etc.
> > > > >
> > > > >
> > > > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]> wrote:
> > > > >
> > > > > > Further development on this today, very little.
> > > > > > I have moved both servers onto Ubuntu 16.04 LTS which means
> going
> > > from
> > > > > perl
> > > > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
> > > > > >
> > > > > > The admin user db did not work after the upgrade so I had to
> empty
> > > the
> > > > > > tables before it would come back online.
> > > > > >
> > > > > > I'm still getting delayed emails and assp shutting down telling
> me
> > > it
> > > > is
> > > > > > unable
> > > > > > to detect any running worker.
> > > > > >
> > > > > > If this goes on much longer the MD will pull the plug and we'll
> > end
> > > up
> > > > > > moving to a third party solution which is not something I want
> but
> > > if
> > > > I
> > > > > > can't fix it I can't defend it :/
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > ------------------
> > > > > > Check out the vibrant tech community on one of the world's most
> > > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > > > _______________________________________________
> > > > > > Assp-test mailing list
> > > > > > [hidden email]
> > > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > > > >
> > > > > >
> > > > >
> > > > > ------------------------------------------------------------
> > > > > ------------------
> > > > > Check out the vibrant tech community on one of the world's most
> > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > > _______________________________________________
> > > > > Assp-test mailing list
> > > > > [hidden email]
> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > > >
> > > > >
> > > > ------------------------------------------------------------
> > > > ------------------
> > > > Check out the vibrant tech community on one of the world's most
> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > _______________________________________________
> > > > Assp-test mailing list
> > > > [hidden email]
> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >
> > > >
> > > >
> > > >
> > > > DISCLAIMER:
> > > > *******************************************************
> > > > This email and any files transmitted with it may be confidential,
> > > legally
> > > > privileged and protected in law and are intended solely for the use
> of
> > > the
> > > >
> > > > individual to whom it is addressed.
> > > > This email was multiple times scanned for viruses. There should be
> no
> > > > known virus in this email!
> > > > *******************************************************
> > > >
> > > >
> > > > ------------------------------------------------------------
> > > > ------------------
> > > > Check out the vibrant tech community on one of the world's most
> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > > _______________________________________________
> > > > Assp-test mailing list
> > > > [hidden email]
> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > > >
> > > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > >
> > >
> > > DISCLAIMER:
> > > *******************************************************
> > > This email and any files transmitted with it may be confidential,
> > legally
> > > privileged and protected in law and are intended solely for the use of
> > the
> > >
> > > individual to whom it is addressed.
> > > This email was multiple times scanned for viruses. There should be no
> > > known virus in this email!
> > > *******************************************************
> > >
> > >
> > > ------------------------------------------------------------
> > > ------------------
> > > Check out the vibrant tech community on one of the world's most
> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > > _______________________________________________
> > > Assp-test mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >
> > >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > *******************************************************
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > *******************************************************
> >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Colin
I can confirm that this number is artificially high quite easily.
I actually have a third ASSP instance. This one is specially configured to
only accept emails from Office 365 and acts as an outbound relay purely to
gather messages for the corpus. This one does not experience the problem
with no running workers being available.

Its stats haven't been reset for a long time and show an average of 2901
per day. Concurrent SMTP sessions since last restart are 0 (4 max).

The last message showing in the logs says:

2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread got connection request
2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread looks up the best
Worker for new connection - 75
2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread will wait (max 30 s)
for the answer of Worker_7 which handles 0 sockets
2016-10-05 14:43:57 [Worker_7] Worker_7 wakes up
2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread freed by idle Worker_7
in 0.005 seconds - got (ok)
2016-10-05 14:43:57 [Worker_7] Info: Worker_7 got connection from
MainThread - 75/75
2016-10-05 14:43:57 [Worker_7] Info: Worker_7 freed Main_Thread - 84

There are no other emails going on at this time. Just the one connection
and it still says 75. This server is firewalled and the only messages it
ever sees are from known Office 365 IP addresses along with the internal
test server that sends 1 message every 60 seconds.

On Wed, Oct 5, 2016 at 2:33 PM, cw <[hidden email]> wrote:

> Hi Thomas,
>
> Thank you for the rundown. The load is exceptionally high at the moment
> because everything backs up every time the mail servers stop. Having said
> that, I've just watched it happen again on one of the servers. It only had
> 7 messages showing in the shutdown_list when it happened. Looking at the
> logs I see:
>
> 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 got connection from
> MainThread - 78/78
> 2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by interrupted
> Worker_4 in 10.758 seconds - got (ok)
> 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 freed Main_Thread - 101
>
> So something is artificially inflating the connection count. My best guess
> ties in with something else I was looking at where a remote server was
> opening two connections that never got answered. It only got through on the
> third connection and that could be replicated each time. I put it down to
> some odd firewall on their end but maybe I was wrong in doing that.
>
> The actual stats on one mailserver are:
>
> Last reset 8 days, 0 hours, 6 mins
> Messages process 165413 (20664 per day)
> SMTP Connections received 113278
>
> The other is:
> Last reset 8 days, 0 hours, 18 mins
> Messaged processed 131720 (16438 per day)
> SMTP Connections received 93088
>
> The stats reset themselves during on of the upgrades of ASSP, not sure why.
>
> Each server is 16 CPU cores, 12GB RAM. DB is on a separate MySQL box
> though MTA and Clam run local. URIBL is disabled. I don't use OCR, DCC or
> Razor. I tried to get OCR working once and all hell broke lose and I
> haven't been back since.
>
> I've been monitoring the performance of the servers all along and not
> seeing any bottlenecks in CPU, RAM or network.When the problem started the
> mail servers had 8 and 10GB RAM respectively (10GB for the one running
> rebuild).
> All the best,
> Colin
>
>
>
> On Wed, Oct 5, 2016 at 2:07 PM, Thomas Eckardt <[hidden email]
> > wrote:
>
>> It seems, that this system is overloaded with active connections.
>>
>> >2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
>> Worker_3 in 2.972 seconds - got (ok)
>>
>> Even this time value is ten times higher than expected.
>>
>> >2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
>>
>> The '170' is the filehandle number in perl (counted from 1.....) - this
>> implies 70 or more active connections.
>>
>> >2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
>> for the answer of Worker_3 which handles 12 sockets
>>
>> This implies the same, if the equal balancing of all connections over all
>> seven workers works correctly.
>>
>> >2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
>> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
>> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
>> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
>> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
>>
>> The processing time itself seems to be normal. SC-Time - is the time
>> required to process one loop (read + queue or read + check + queue) for
>> one active connection.
>>
>> Assuming an average of 4 second life time per connection - the workload
>> per day would be 1.5 million (70 / 4 *3600 *24) connections based on the
>> provided output.
>> Configured to an ISP mode the avg. maximum for a single assp instance is
>> 800.000 connections. To be able to handle workload peeks, the
>> configuration should not allow more than 400.000 connection a day.
>>
>> ISP mode:
>> - 16 CPU cores
>> - 16 Workers
>> - 24 GB RAM
>> - dedicated systems for MTA, enterprise database, ClamAV
>> - HMM and spamDB holded unshared in RAM
>> - high performance DNS servers
>> - limited DNS usage (disable most extensive DNS-using checks - eg. URIBL)
>> - no OCR
>> - no DCC
>> - no Razor2
>>
>> >2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
>> Worker_3 in 31.940 seconds - got (ok)
>>
>> This time value is near the end of the line. At this time a small count of
>> additionally connections can lead in to an assp shutdown, because the
>> workers don't accept new connections.
>>
>> Thomas
>>
>>
>>
>> Von:    cw <[hidden email]>
>> An:     ASSP development mailing list <[hidden email]>
>> Datum:  05.10.2016 13:15
>> Betreff:        Re: [Assp-test] unable to detect any running worker
>>
>>
>>
>> Thanks.
>>
>> I've had both servers come up against unable to detect any running worker
>> since clearing out all the files suggested. So I'm getting 16279 running
>> now.
>>
>> I noticed the startup with those files removed was really quick, starting
>> back up the second time took several minutes so presumably reading those
>> files during startup takes a little while versus creating them fresh.
>>
>> I've caught one already and traced it through the new logs:
>>
>> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got connection request
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up the best
>> Worker for new connection - 73
>> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt worker Worker_3
>> (12) for new connection
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread interrupted Worker_3
>> (12) to submit the connection
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
>> for the answer of Worker_3 which handles 12 sockets
>> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3: 0.0462169647216797
>> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection from
>> MainThread - 73/73
>> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
>> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
>> Worker_3 in 2.972 seconds - got (ok)
>> 2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
>> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
>> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016 11:47:41
>> +0100
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
>> 0.0296478271484375
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
>> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
>> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
>> Worker_3 in 31.940 seconds - got (ok)
>>
>> In this case, it looks like the connection ended without any actual data.
>> There is lots of activity reported by Worker_3 in between 11:47:43 and
>> 11:48:13 but this all pertains to other connections that were already in
>> progress.
>>
>> That is a bit different to the earlier one where a message was received
>> and
>> the message completed within the 30s window.
>>
>> I'm not seeing anything to help me figure out why though and I don't want
>> to simply post a big excerpt of the maillog.txt.
>>
>> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt
>> <[hidden email]>
>> wrote:
>>
>> > >I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
>> > it
>> > hasn't made its way live yet.
>> >
>> > Sorry, my background CVS sync was not running - update is done.
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> > Von:    cw <[hidden email]>
>> > An:     ASSP development mailing list <[hidden email]>
>> > Datum:  05.10.2016 11:48
>> > Betreff:        Re: [Assp-test] unable to detect any running worker
>> >
>> >
>> >
>> > Thank you Thomas.
>> >
>> > useDB4IntCache - already set to off
>> > I've set WorkerLog to diagnostic and done the other steps.
>> > I don't have anything in CorrectASSPcfg.pm so as part of the
>> > troubleshooting I have previously deleted it and downloaded a fresh copy
>> > from SourceForge.
>> >
>> > I have to say that ASSP started up within seconds after clearing those
>> > files out - it normally takes several minutes and has always done.
>> >
>> > I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
>> it
>> > hasn't made its way live yet.
>> >
>> > On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
>> > <[hidden email]>
>> > wrote:
>> >
>> > > I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
>> > version
>> > > shows some more information, if 'WorkerLog' is set to diagnostic.
>> > >
>> > > Thomas
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Von:    cw <[hidden email]>
>> > > An:     ASSP development mailing list
>> <[hidden email]>
>> > > Datum:  05.10.2016 10:10
>> > > Betreff:        Re: [Assp-test] unable to detect any running worker
>> > >
>> > >
>> > >
>> > > Hi Thomas,
>> > >
>> > > Thanks for chipping in. All modules are installed by running the
>> latest
>> > > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
>> > requires
>> > > a minor edit to the code to get it to build. So that module was
>> > installed
>> > > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
>> > > make clean
>> > > perl Makefile.PL
>> > > make
>> > > make test
>> > > make install
>> > >
>> > > I run cpan-outdated -p|cpanm from time to time in order to keep
>> modules
>> > up
>> > > to date as well so things don't stay stuck on old versions.
>> > >
>> > > Everything that is installed now is a completely fresh build as the
>> > > Upgrade
>> > > to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore
>> had
>> > to
>> > > be installed from scratch.
>> > >
>> > > I bypassed the issue by truncating the tables and setting up the users
>> > > again. With there only being one user it was easier to do that than
>> muck
>> > > about with it - especially with the other bigger issue at hand.
>> > >
>> > > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
>> > > <[hidden email]>
>> > > wrote:
>> > >
>> > > > >to replace the corrupted encrypted
>> > > > strings with the correct values
>> > > >
>> > > > Was the 'Crypt::GOST' module from the SF download page at the old
>> assp
>> > > > instance?
>> > > > If it was, did you install the 'Crypt::GOST' module from the SF
>> > download
>> > > > page, before you started the new assp instance?
>> > > >
>> > > > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
>> > > > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>> > > >
>> > > > Thomas
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Von:    cw <[hidden email]>
>> > > > An:     ASSP development mailing list
>> > <[hidden email]>
>> > > > Datum:  05.10.2016 09:52
>> > > > Betreff:        Re: [Assp-test] unable to detect any running worker
>> > > >
>> > > >
>> > > >
>> > > > Cheers for the reply.
>> > > >
>> > > > I don't think backups are an option seen as I've moved completely
>> from
>> > > > Ubuntu 14.04 to Ubuntu 16.04.
>> > > >
>> > > > Also, this issue has been around for months. It was causing a
>> handful
>> > of
>> > > > shutdowns a week with the occasional spat of more frequent
>> shutdowns.
>> > It
>> > > > is
>> > > > entirely possible that the errors are behaviour related and nothing
>> to
>> > > do
>> > > > with the upgrade and that current email behaviour is triggering it
>> big
>> > > > style.
>> > > >
>> > > > I'm not convinced though, unfortunately I'm not convinced of
>> anything
>> > > else
>> > > > hence not having much to go on. I can't see any consistencies in the
>> > > > behaviour leading up to the events.
>> > > >
>> > > > I don't think it is database related. It happened on one of the mail
>> > > > servers during the upgrade to 16.04 when ASSP had just started but I
>> > had
>> > > > not yet got into the web interface to replace the corrupted
>> encrypted
>> > > > strings with the correct values so all database connections were in
>> > > error.
>> > > >
>> > > > The problem has already started again this morning so I can see this
>> > > being
>> > > > another fun day that either leads to a fix or having to put
>> something
>> > > else
>> > > > in place.
>> > > >
>> > > > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]> wrote:
>> > > >
>> > > > > I've been reading here, but I haven't had anything to suggest. All
>> > > > seems
>> > > > > quite odd if it was working prior to upgrading and downgrading
>> > didn't
>> > > > work.
>> > > > >
>> > > > >
>> > > > > Could you spin up a backup of the installation after copying the
>> > > current
>> > > > > data?  Sure you'd have an older corpus, but I'd think you could
>> add
>> > > the
>> > > > new
>> > > > > files if necessary, manually replace whitelist etc.
>> > > > >
>> > > > >
>> > > > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]>
>> wrote:
>> > > > >
>> > > > > > Further development on this today, very little.
>> > > > > > I have moved both servers onto Ubuntu 16.04 LTS which means
>> going
>> > > from
>> > > > > perl
>> > > > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
>> > > > > >
>> > > > > > The admin user db did not work after the upgrade so I had to
>> empty
>> > > the
>> > > > > > tables before it would come back online.
>> > > > > >
>> > > > > > I'm still getting delayed emails and assp shutting down telling
>> me
>> > > it
>> > > > is
>> > > > > > unable
>> > > > > > to detect any running worker.
>> > > > > >
>> > > > > > If this goes on much longer the MD will pull the plug and we'll
>> > end
>> > > up
>> > > > > > moving to a third party solution which is not something I want
>> but
>> > > if
>> > > > I
>> > > > > > can't fix it I can't defend it :/
>> > > > > >
>> > > > > > ------------------------------------------------------------
>> > > > > > ------------------
>> > > > > > Check out the vibrant tech community on one of the world's most
>> > > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > > > _______________________________________________
>> > > > > > Assp-test mailing list
>> > > > > > [hidden email]
>> > > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > > > >
>> > > > > >
>> > > > >
>> > > > > ------------------------------------------------------------
>> > > > > ------------------
>> > > > > Check out the vibrant tech community on one of the world's most
>> > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > > _______________________________________________
>> > > > > Assp-test mailing list
>> > > > > [hidden email]
>> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > > >
>> > > > >
>> > > > ------------------------------------------------------------
>> > > > ------------------
>> > > > Check out the vibrant tech community on one of the world's most
>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > _______________________________________________
>> > > > Assp-test mailing list
>> > > > [hidden email]
>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > DISCLAIMER:
>> > > > *******************************************************
>> > > > This email and any files transmitted with it may be confidential,
>> > > legally
>> > > > privileged and protected in law and are intended solely for the use
>> of
>> > > the
>> > > >
>> > > > individual to whom it is addressed.
>> > > > This email was multiple times scanned for viruses. There should be
>> no
>> > > > known virus in this email!
>> > > > *******************************************************
>> > > >
>> > > >
>> > > > ------------------------------------------------------------
>> > > > ------------------
>> > > > Check out the vibrant tech community on one of the world's most
>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > _______________________________________________
>> > > > Assp-test mailing list
>> > > > [hidden email]
>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > >
>> > > >
>> > > ------------------------------------------------------------
>> > > ------------------
>> > > Check out the vibrant tech community on one of the world's most
>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > _______________________________________________
>> > > Assp-test mailing list
>> > > [hidden email]
>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > >
>> > >
>> > >
>> > >
>> > > DISCLAIMER:
>> > > *******************************************************
>> > > This email and any files transmitted with it may be confidential,
>> > legally
>> > > privileged and protected in law and are intended solely for the use of
>> > the
>> > >
>> > > individual to whom it is addressed.
>> > > This email was multiple times scanned for viruses. There should be no
>> > > known virus in this email!
>> > > *******************************************************
>> > >
>> > >
>> > > ------------------------------------------------------------
>> > > ------------------
>> > > Check out the vibrant tech community on one of the world's most
>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > _______________________________________________
>> > > Assp-test mailing list
>> > > [hidden email]
>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > >
>> > >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> >
>> >
>> > DISCLAIMER:
>> > *******************************************************
>> > This email and any files transmitted with it may be confidential,
>> legally
>> > privileged and protected in law and are intended solely for the use of
>> the
>> >
>> > individual to whom it is addressed.
>> > This email was multiple times scanned for viruses. There should be no
>> > known virus in this email!
>> > *******************************************************
>> >
>> >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential, legally
>> privileged and protected in law and are intended solely for the use of the
>>
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
Reply | Threaded
Open this post in threaded view
|

Re: unable to detect any running worker

Thomas Eckardt/eck
What are your setting for :

ThreadCycleTime
EnableHighPerformance

Thomas



Von:    cw <[hidden email]>
An:     ASSP development mailing list <[hidden email]>
Datum:  05.10.2016 15:51
Betreff:        Re: [Assp-test] unable to detect any running worker



I can confirm that this number is artificially high quite easily.
I actually have a third ASSP instance. This one is specially configured to
only accept emails from Office 365 and acts as an outbound relay purely to
gather messages for the corpus. This one does not experience the problem
with no running workers being available.

Its stats haven't been reset for a long time and show an average of 2901
per day. Concurrent SMTP sessions since last restart are 0 (4 max).

The last message showing in the logs says:

2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread got connection request
2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread looks up the best
Worker for new connection - 75
2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread will wait (max 30 s)
for the answer of Worker_7 which handles 0 sockets
2016-10-05 14:43:57 [Worker_7] Worker_7 wakes up
2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread freed by idle Worker_7
in 0.005 seconds - got (ok)
2016-10-05 14:43:57 [Worker_7] Info: Worker_7 got connection from
MainThread - 75/75
2016-10-05 14:43:57 [Worker_7] Info: Worker_7 freed Main_Thread - 84

There are no other emails going on at this time. Just the one connection
and it still says 75. This server is firewalled and the only messages it
ever sees are from known Office 365 IP addresses along with the internal
test server that sends 1 message every 60 seconds.

On Wed, Oct 5, 2016 at 2:33 PM, cw <[hidden email]> wrote:

> Hi Thomas,
>
> Thank you for the rundown. The load is exceptionally high at the moment
> because everything backs up every time the mail servers stop. Having
said
> that, I've just watched it happen again on one of the servers. It only
had

> 7 messages showing in the shutdown_list when it happened. Looking at the
> logs I see:
>
> 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 got connection from
> MainThread - 78/78
> 2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by interrupted
> Worker_4 in 10.758 seconds - got (ok)
> 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 freed Main_Thread - 101
>
> So something is artificially inflating the connection count. My best
guess
> ties in with something else I was looking at where a remote server was
> opening two connections that never got answered. It only got through on
the
> third connection and that could be replicated each time. I put it down
to

> some odd firewall on their end but maybe I was wrong in doing that.
>
> The actual stats on one mailserver are:
>
> Last reset 8 days, 0 hours, 6 mins
> Messages process 165413 (20664 per day)
> SMTP Connections received 113278
>
> The other is:
> Last reset 8 days, 0 hours, 18 mins
> Messaged processed 131720 (16438 per day)
> SMTP Connections received 93088
>
> The stats reset themselves during on of the upgrades of ASSP, not sure
why.
>
> Each server is 16 CPU cores, 12GB RAM. DB is on a separate MySQL box
> though MTA and Clam run local. URIBL is disabled. I don't use OCR, DCC
or
> Razor. I tried to get OCR working once and all hell broke lose and I
> haven't been back since.
>
> I've been monitoring the performance of the servers all along and not
> seeing any bottlenecks in CPU, RAM or network.When the problem started
the
> mail servers had 8 and 10GB RAM respectively (10GB for the one running
> rebuild).
> All the best,
> Colin
>
>
>
> On Wed, Oct 5, 2016 at 2:07 PM, Thomas Eckardt
<[hidden email]
> > wrote:
>
>> It seems, that this system is overloaded with active connections.
>>
>> >2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by
interrupted

>> Worker_3 in 2.972 seconds - got (ok)
>>
>> Even this time value is ten times higher than expected.
>>
>> >2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
>>
>> The '170' is the filehandle number in perl (counted from 1.....) - this
>> implies 70 or more active connections.
>>
>> >2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30
s)
>> for the answer of Worker_3 which handles 12 sockets
>>
>> This implies the same, if the equal balancing of all connections over
all

>> seven workers works correctly.
>>
>> >2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
>> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
>> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
>> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
>> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
>>
>> The processing time itself seems to be normal. SC-Time - is the time
>> required to process one loop (read + queue or read + check + queue) for
>> one active connection.
>>
>> Assuming an average of 4 second life time per connection - the workload
>> per day would be 1.5 million (70 / 4 *3600 *24) connections based on
the
>> provided output.
>> Configured to an ISP mode the avg. maximum for a single assp instance
is

>> 800.000 connections. To be able to handle workload peeks, the
>> configuration should not allow more than 400.000 connection a day.
>>
>> ISP mode:
>> - 16 CPU cores
>> - 16 Workers
>> - 24 GB RAM
>> - dedicated systems for MTA, enterprise database, ClamAV
>> - HMM and spamDB holded unshared in RAM
>> - high performance DNS servers
>> - limited DNS usage (disable most extensive DNS-using checks - eg.
URIBL)
>> - no OCR
>> - no DCC
>> - no Razor2
>>
>> >2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by
interrupted
>> Worker_3 in 31.940 seconds - got (ok)
>>
>> This time value is near the end of the line. At this time a small count
of

>> additionally connections can lead in to an assp shutdown, because the
>> workers don't accept new connections.
>>
>> Thomas
>>
>>
>>
>> Von:    cw <[hidden email]>
>> An:     ASSP development mailing list <[hidden email]>
>> Datum:  05.10.2016 13:15
>> Betreff:        Re: [Assp-test] unable to detect any running worker
>>
>>
>>
>> Thanks.
>>
>> I've had both servers come up against unable to detect any running
worker
>> since clearing out all the files suggested. So I'm getting 16279
running
>> now.
>>
>> I noticed the startup with those files removed was really quick,
starting
>> back up the second time took several minutes so presumably reading
those
>> files during startup takes a little while versus creating them fresh.
>>
>> I've caught one already and traced it through the new logs:
>>
>> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got connection
request
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up the best
>> Worker for new connection - 73
>> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt worker
Worker_3
>> (12) for new connection
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread interrupted
Worker_3
>> (12) to submit the connection
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30
s)
>> for the answer of Worker_3 which handles 12 sockets
>> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3: 0.0462169647216797
>> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection from
>> MainThread - 73/73
>> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
>> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by
interrupted

>> Worker_3 in 2.972 seconds - got (ok)
>> 2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
>> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
>> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016 11:47:41
>> +0100
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
>> 0.0296478271484375
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
>> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
>> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by
interrupted
>> Worker_3 in 31.940 seconds - got (ok)
>>
>> In this case, it looks like the connection ended without any actual
data.
>> There is lots of activity reported by Worker_3 in between 11:47:43 and
>> 11:48:13 but this all pertains to other connections that were already
in
>> progress.
>>
>> That is a bit different to the earlier one where a message was received
>> and
>> the message completed within the 30s window.
>>
>> I'm not seeing anything to help me figure out why though and I don't
want
>> to simply post a big excerpt of the maillog.txt.
>>
>> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt
>> <[hidden email]>
>> wrote:
>>
>> > >I looked at SF but only see 16275 in test (updated 3 days ago) so
maybe

>> > it
>> > hasn't made its way live yet.
>> >
>> > Sorry, my background CVS sync was not running - update is done.
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> > Von:    cw <[hidden email]>
>> > An:     ASSP development mailing list
<[hidden email]>

>> > Datum:  05.10.2016 11:48
>> > Betreff:        Re: [Assp-test] unable to detect any running worker
>> >
>> >
>> >
>> > Thank you Thomas.
>> >
>> > useDB4IntCache - already set to off
>> > I've set WorkerLog to diagnostic and done the other steps.
>> > I don't have anything in CorrectASSPcfg.pm so as part of the
>> > troubleshooting I have previously deleted it and downloaded a fresh
copy
>> > from SourceForge.
>> >
>> > I have to say that ASSP started up within seconds after clearing
those
>> > files out - it normally takes several minutes and has always done.
>> >
>> > I looked at SF but only see 16275 in test (updated 3 days ago) so
maybe

>> it
>> > hasn't made its way live yet.
>> >
>> > On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
>> > <[hidden email]>
>> > wrote:
>> >
>> > > I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
>> > version
>> > > shows some more information, if 'WorkerLog' is set to diagnostic.
>> > >
>> > > Thomas
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Von:    cw <[hidden email]>
>> > > An:     ASSP development mailing list
>> <[hidden email]>
>> > > Datum:  05.10.2016 10:10
>> > > Betreff:        Re: [Assp-test] unable to detect any running worker
>> > >
>> > >
>> > >
>> > > Hi Thomas,
>> > >
>> > > Thanks for chipping in. All modules are installed by running the
>> latest
>> > > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
>> > requires
>> > > a minor edit to the code to get it to build. So that module was
>> > installed
>> > > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
>> > > make clean
>> > > perl Makefile.PL
>> > > make
>> > > make test
>> > > make install
>> > >
>> > > I run cpan-outdated -p|cpanm from time to time in order to keep
>> modules
>> > up
>> > > to date as well so things don't stay stuck on old versions.
>> > >
>> > > Everything that is installed now is a completely fresh build as the
>> > > Upgrade
>> > > to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore
>> had
>> > to
>> > > be installed from scratch.
>> > >
>> > > I bypassed the issue by truncating the tables and setting up the
users

>> > > again. With there only being one user it was easier to do that than
>> muck
>> > > about with it - especially with the other bigger issue at hand.
>> > >
>> > > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
>> > > <[hidden email]>
>> > > wrote:
>> > >
>> > > > >to replace the corrupted encrypted
>> > > > strings with the correct values
>> > > >
>> > > > Was the 'Crypt::GOST' module from the SF download page at the old
>> assp
>> > > > instance?
>> > > > If it was, did you install the 'Crypt::GOST' module from the SF
>> > download
>> > > > page, before you started the new assp instance?
>> > > >
>> > > > <a href="https://sourceforge.net/projects/assp/files/ASSP%20V2%">https://sourceforge.net/projects/assp/files/ASSP%20V2%
>> > > > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>> > > >
>> > > > Thomas
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Von:    cw <[hidden email]>
>> > > > An:     ASSP development mailing list
>> > <[hidden email]>
>> > > > Datum:  05.10.2016 09:52
>> > > > Betreff:        Re: [Assp-test] unable to detect any running
worker

>> > > >
>> > > >
>> > > >
>> > > > Cheers for the reply.
>> > > >
>> > > > I don't think backups are an option seen as I've moved completely
>> from
>> > > > Ubuntu 14.04 to Ubuntu 16.04.
>> > > >
>> > > > Also, this issue has been around for months. It was causing a
>> handful
>> > of
>> > > > shutdowns a week with the occasional spat of more frequent
>> shutdowns.
>> > It
>> > > > is
>> > > > entirely possible that the errors are behaviour related and
nothing
>> to
>> > > do
>> > > > with the upgrade and that current email behaviour is triggering
it
>> big
>> > > > style.
>> > > >
>> > > > I'm not convinced though, unfortunately I'm not convinced of
>> anything
>> > > else
>> > > > hence not having much to go on. I can't see any consistencies in
the
>> > > > behaviour leading up to the events.
>> > > >
>> > > > I don't think it is database related. It happened on one of the
mail
>> > > > servers during the upgrade to 16.04 when ASSP had just started
but I
>> > had
>> > > > not yet got into the web interface to replace the corrupted
>> encrypted
>> > > > strings with the correct values so all database connections were
in
>> > > error.
>> > > >
>> > > > The problem has already started again this morning so I can see
this
>> > > being
>> > > > another fun day that either leads to a fix or having to put
>> something
>> > > else
>> > > > in place.
>> > > >
>> > > > On Wed, Oct 5, 2016 at 4:34 AM, K Post <[hidden email]>
wrote:
>> > > >
>> > > > > I've been reading here, but I haven't had anything to suggest.
All
>> > > > seems
>> > > > > quite odd if it was working prior to upgrading and downgrading
>> > didn't
>> > > > work.
>> > > > >
>> > > > >
>> > > > > Could you spin up a backup of the installation after copying
the

>> > > current
>> > > > > data?  Sure you'd have an older corpus, but I'd think you could
>> add
>> > > the
>> > > > new
>> > > > > files if necessary, manually replace whitelist etc.
>> > > > >
>> > > > >
>> > > > > On Tue, Oct 4, 2016 at 5:48 PM, cw <[hidden email]>
>> wrote:
>> > > > >
>> > > > > > Further development on this today, very little.
>> > > > > > I have moved both servers onto Ubuntu 16.04 LTS which means
>> going
>> > > from
>> > > > > perl
>> > > > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
>> > > > > >
>> > > > > > The admin user db did not work after the upgrade so I had to
>> empty
>> > > the
>> > > > > > tables before it would come back online.
>> > > > > >
>> > > > > > I'm still getting delayed emails and assp shutting down
telling
>> me
>> > > it
>> > > > is
>> > > > > > unable
>> > > > > > to detect any running worker.
>> > > > > >
>> > > > > > If this goes on much longer the MD will pull the plug and
we'll
>> > end
>> > > up
>> > > > > > moving to a third party solution which is not something I
want
>> but
>> > > if
>> > > > I
>> > > > > > can't fix it I can't defend it :/
>> > > > > >
>> > > > > > ------------------------------------------------------------
>> > > > > > ------------------
>> > > > > > Check out the vibrant tech community on one of the world's
most

>> > > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > > > _______________________________________________
>> > > > > > Assp-test mailing list
>> > > > > > [hidden email]
>> > > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > > > >
>> > > > > >
>> > > > >
>> > > > > ------------------------------------------------------------
>> > > > > ------------------
>> > > > > Check out the vibrant tech community on one of the world's most
>> > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > > _______________________________________________
>> > > > > Assp-test mailing list
>> > > > > [hidden email]
>> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > > >
>> > > > >
>> > > > ------------------------------------------------------------
>> > > > ------------------
>> > > > Check out the vibrant tech community on one of the world's most
>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > _______________________________________________
>> > > > Assp-test mailing list
>> > > > [hidden email]
>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > DISCLAIMER:
>> > > > *******************************************************
>> > > > This email and any files transmitted with it may be confidential,
>> > > legally
>> > > > privileged and protected in law and are intended solely for the
use
>> of
>> > > the
>> > > >
>> > > > individual to whom it is addressed.
>> > > > This email was multiple times scanned for viruses. There should
be

>> no
>> > > > known virus in this email!
>> > > > *******************************************************
>> > > >
>> > > >
>> > > > ------------------------------------------------------------
>> > > > ------------------
>> > > > Check out the vibrant tech community on one of the world's most
>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > > _______________________________________________
>> > > > Assp-test mailing list
>> > > > [hidden email]
>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > > >
>> > > >
>> > > ------------------------------------------------------------
>> > > ------------------
>> > > Check out the vibrant tech community on one of the world's most
>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > _______________________________________________
>> > > Assp-test mailing list
>> > > [hidden email]
>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > >
>> > >
>> > >
>> > >
>> > > DISCLAIMER:
>> > > *******************************************************
>> > > This email and any files transmitted with it may be confidential,
>> > legally
>> > > privileged and protected in law and are intended solely for the use
of
>> > the
>> > >
>> > > individual to whom it is addressed.
>> > > This email was multiple times scanned for viruses. There should be
no

>> > > known virus in this email!
>> > > *******************************************************
>> > >
>> > >
>> > > ------------------------------------------------------------
>> > > ------------------
>> > > Check out the vibrant tech community on one of the world's most
>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > > _______________________________________________
>> > > Assp-test mailing list
>> > > [hidden email]
>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>> > >
>> > >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> >
>> >
>> > DISCLAIMER:
>> > *******************************************************
>> > This email and any files transmitted with it may be confidential,
>> legally
>> > privileged and protected in law and are intended solely for the use
of

>> the
>> >
>> > individual to whom it is addressed.
>> > This email was multiple times scanned for viruses. There should be no
>> > known virus in this email!
>> > *******************************************************
>> >
>> >
>> > ------------------------------------------------------------
>> > ------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Assp-test mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>> >
>> >
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential,
legally
>> privileged and protected in law and are intended solely for the use of
the

>>
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-test
12