ClamAv Up / ClamAv Down

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

ClamAv Up / ClamAv Down

Raynaud Alexandre
Hi list,
Since we receive many emails with virus (cryptowall.....), we have
strengthened our clamav with numerous unofficial signatures
(https://github.com/extremeshok/clamav-unofficial-sigs)
Clamav is of course installed in the same box as assp (ASSP version
2.4.7(16004)) and is configured with local socket.
Actually, clamd check databases every hour and it takes 25s  to completly
reload databases (including new databases downloaded by freshclam and
clamav-unofficial-sigs script). During this time i guess clamd is not
accessible.
The big concern we have is that if ASSP check clamd status during this
"reloading time" it consider clamd daemon to be down (that's normal) but
ASSP takes long time to consider clamd daemon accessible again, more time
than the 25s needed by clamd to (force) reload its databases.
So in a full 24 hours, the total reloading database times of clamd is :
25s*24 : 600s or 10 minutes(considering that SelfCheck is 3600s in
clamd.conf and it takes 25s each time clamd reload its 6200316 signatures)

Here in clamd.log we can see time needed by clamd to reload databases every
hours (24 hours log):
Tue Feb 16 00:49:45 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 00:49:46 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 00:50:09 2016 -> Database correctly reloaded (6201561 signatures)
Tue Feb 16 01:57:28 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 01:57:29 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 01:57:53 2016 -> Database correctly reloaded (6201590 signatures)
Tue Feb 16 02:59:41 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 02:59:42 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 03:00:05 2016 -> Database correctly reloaded (6193678 signatures)
Tue Feb 16 04:07:24 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 04:07:25 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 04:07:48 2016 -> Database correctly reloaded (6193685 signatures)
Tue Feb 16 05:21:38 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 05:21:39 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 05:22:03 2016 -> Database correctly reloaded (6193690 signatures)
Tue Feb 16 06:26:01 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 06:26:02 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 06:26:26 2016 -> Database correctly reloaded (6193692 signatures)
Tue Feb 16 06:26:26 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 06:26:50 2016 -> Database correctly reloaded (6193692 signatures)
Tue Feb 16 07:29:54 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 07:29:54 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 07:30:18 2016 -> Database correctly reloaded (6193692 signatures)
Tue Feb 16 08:31:10 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 08:31:11 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 08:31:34 2016 -> Database correctly reloaded (6193702 signatures)
Tue Feb 16 09:32:33 2016 -> SelfCheck: Database status OK.
Tue Feb 16 10:32:40 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 10:32:41 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 10:33:05 2016 -> Database correctly reloaded (6193698 signatures)
Tue Feb 16 11:33:39 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 11:33:40 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 11:34:03 2016 -> Database correctly reloaded (6193682 signatures)
Tue Feb 16 12:35:20 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 12:35:21 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 12:35:45 2016 -> Database correctly reloaded (6193657 signatures)
Tue Feb 16 13:43:11 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 13:43:12 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 13:43:36 2016 -> Database correctly reloaded (6193664 signatures)
Tue Feb 16 14:43:49 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 14:43:50 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 14:44:13 2016 -> Database correctly reloaded (6193694 signatures)
Tue Feb 16 15:44:56 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 15:44:57 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 15:45:21 2016 -> Database correctly reloaded (6193743 signatures)
Tue Feb 16 16:45:28 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 16:45:28 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 16:45:52 2016 -> Database correctly reloaded (6207843 signatures)
Tue Feb 16 17:46:12 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 17:46:13 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 17:46:37 2016 -> Database correctly reloaded (6207882 signatures)
Tue Feb 16 18:26:32 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 18:26:56 2016 -> Database correctly reloaded (6210992 signatures)
Tue Feb 16 19:28:53 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 19:28:54 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 19:29:18 2016 -> Database correctly reloaded (6211016 signatures)
Tue Feb 16 20:35:47 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 20:35:48 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 20:36:11 2016 -> Database correctly reloaded (6211041 signatures)
Tue Feb 16 21:57:20 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 21:57:21 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 21:57:45 2016 -> Database correctly reloaded (6211090 signatures)
Tue Feb 16 23:25:21 2016 -> SelfCheck: Database modification detected.
Forcing reload.
Tue Feb 16 23:25:22 2016 -> Reading databases from /var/lib/clamav
Tue Feb 16 23:25:46 2016 -> Database correctly reloaded (6211092 signatures)


But in assp log file i can see that ASSP considers clamd down for a far
longer time.
Here is a grep "Clam" from yesterday assp maillog :
fév-16-16 01:57:33 [Worker_1] ClamAv Down
fév-16-16 02:12:46 [Worker_1] ClamAv Up --> 0:15:13 downtime
fév-16-16 02:59:46 [Worker_1] ClamAv Down
fév-16-16 03:01:01 [Worker_1] ClamAv Up --> 0:01:15 downtime
fév-16-16 05:21:43 [Worker_1] ClamAv Down
fév-16-16 05:26:09 [Worker_1] ClamAv Up --> 0:04:26 downtime
fév-16-16 07:29:59 [Worker_1] ClamAv Down
fév-16-16 07:33:37 [Worker_1] ClamAv Up --> 0:03:38 downtime
fév-16-16 08:31:15 [Worker_1] ClamAv Down
fév-16-16 08:32:04 [Worker_1] ClamAv Up --> 0:00:49 downtime
fév-16-16 10:32:55 [Worker_1] ClamAv Down
fév-16-16 10:35:01 [Worker_1] ClamAv Up --> 0:02:06 downtime
fév-16-16 11:33:54 [Worker_1] ClamAv Down
fév-16-16 11:34:03 [Worker_3] ClamAv Up --> 0:00:09 downtime

fév-16-16 11:34:03 [Worker_2] ClamAv Up
fév-16-16 12:35:33 [Worker_1] ClamAv Down
fév-16-16 12:35:54 [Worker_1] ClamAv Up --> 0:00:21 downtime
fév-16-16 13:43:24 [Worker_1] ClamAv Down
fév-16-16 13:44:46 [Worker_1] ClamAv Up --> 0:01:22 downtime
fév-16-16 14:44:02 [Worker_1] ClamAv Down
fév-16-16 14:44:16 [Worker_1] ClamAv Up --> 0:00:14 downtime
fév-16-16 15:45:09 [Main_Thread] ClamAv Down
fév-16-16 15:45:21 [Main_Thread] ClamAv Up --> 0:00:12 downtime
fév-16-16 16:45:41 [Worker_1] ClamAv Down
fév-16-16 16:48:09 [Worker_1] ClamAv Up --> 0:02:28 downtime
fév-16-16 17:46:25 [Worker_1] ClamAv Down
fév-16-16 17:47:59 [Worker_1] ClamAv Up --> 0:01:34 downtime
fév-16-16 19:29:06 [Worker_1] ClamAv Down
fév-16-16 19:29:58 [Worker_1] ClamAv Up --> 0:00:52 downtime
fév-16-16 20:36:00 [Worker_1] ClamAv Down
fév-16-16 20:40:00 [Worker_1] ClamAv Up --> 0:04:00 downtime
fév-16-16 21:57:33 [Worker_1] ClamAv Down
fév-16-16 22:07:26 [Worker_1] ClamAv Up --> 0:09:53 downtime
fév-16-16 23:25:34 [Worker_1] ClamAv Down
fév-16-16 23:40:07 [Worker_1] ClamAv Up --> 0:14:33 downtime

So assp consider clamd daemon down for 1:03:05...................

I have tried to increase ClamAV Timeout (ClamAVtimeout) to 30s but it
doesn't change the behaviour of assp......

This is a big concern for us and we really need antivirus functionalities.

Is there a way to force ASSP to imperatively wait clamd to be up before
proceed any emails (even if it slows mail delivery) and a trick to make it
realise clamd is up as soon clamd finished reload its database (25s)?

I would appreciate your help, advice and opinion.

Regards,
Alexandre RAYNAUD
MAIRIE DE SALLANCHES  

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Assp-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-user
Reply | Threaded
Open this post in threaded view
|

Re: ClamAv Up / ClamAv Down

GrayHat
:: On Wed, 17 Feb 2016 12:27:58 +0100
::
<[hidden email]> ::
"Raynaud Alexandre" <[hidden email]> wrote:


> Is there a way to force ASSP to imperatively wait clamd to be up
> before proceed any emails (even if it slows mail delivery) and a
> trick to make it realise clamd is up as soon clamd finished reload
> its database (25s)?

No, and it doesn't really make sense; the issue is due to the way
ClamAV reloads signatures and not to the way ASSP checks for its
availability; making ASSP dependent on clamd to work is a nonsense,
clamav is just ONE of the filters used.


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Assp-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-user
Reply | Threaded
Open this post in threaded view
|

Re: ClamAv Up / ClamAv Down

GrayHat
:: On Wed, 17 Feb 2016 12:55:08 +0100
:: <[hidden email]>
:: Grayhat <[hidden email]> wrote:

> :: On Wed, 17 Feb 2016 12:27:58 +0100
> ::
> <[hidden email]> ::
> "Raynaud Alexandre" <[hidden email]> wrote:
>
>
> > Is there a way to force ASSP to imperatively wait clamd to be up
> > before proceed any emails (even if it slows mail delivery) and a
> > trick to make it realise clamd is up as soon clamd finished reload
> > its database (25s)?  
 
> No, and it doesn't really make sense; the issue is due to the way
> ClamAV reloads signatures and not to the way ASSP checks for its
> availability; making ASSP dependent on clamd to work is a nonsense,
> clamav is just ONE of the filters used.

oh and just in case, the signatures ARE NOT provided by "extremeshok"
but by Steve from SaneSecurity

http://sanesecurity.com/usage/signatures/ 

and he, correctly, says to *avoid* downloading the signatures a lot of
times per hour since not only it doesn't make sense, but overloads the
servers so, if you download the signature too frequently your IP will
be banned as clearly explained here

http://sanesecurity.com/usage/linux-scripts/

The mirrors reserve the right to block your IP address, if you are
downloading too many times per hour or are abusing their
servers/bandwidth in any way.


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Assp-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-user
Reply | Threaded
Open this post in threaded view
|

Re: ClamAv Up / ClamAv Down

GrayHat
:: On Wed, 17 Feb 2016 13:01:06 +0100
:: <[hidden email]>
:: Grayhat <[hidden email]> wrote:

> :: On Wed, 17 Feb 2016 12:55:08 +0100

> > No, and it doesn't really make sense; the issue is due to the way
> > ClamAV reloads signatures and not to the way ASSP checks for its
> > availability; making ASSP dependent on clamd to work is a nonsense,
> > clamav is just ONE of the filters used.  

and since we're at it; ASSP uses a whole arsenal of methods to detect
spam/notspam messages, and being a "frontline" filter MUST be as fast
as possible, this excludes the use of "longer than usual" checks; then,
if you need to perform some more costly (in terms of time/CPU/other...)
kind of filtering, you'd better consider a "second stage" filter, that
is a filter applied on the pipeline which, at end, brings to the end
user mailbox; in your case, you may consider using an instance of an
SMTP "router" (be it postfix, sendmail or whatever) which may perform
further checking on the incoming messages (already accepted by ASSP)
and "tag" them so that they'll be moved to the user's "spam" (or, in
case it exists, "virus") folder; but "forcing ASSP to wait for clamd"
is a very BAD idea, the next one would be forcing ASSP to wait a lot of
time for DNSBL/URIBL answers or pretending to have it run some deep and
time consuming checks on each and every incoming email and, at the very
same time, complaining for timeout, slowdowns and so on :P


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Assp-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-user
Reply | Threaded
Open this post in threaded view
|

Re: ClamAv Up / ClamAv Down

Robert K Coffman Jr. -Info From Data Corp.
In reply to this post by Raynaud Alexandre
 > On 2/17/2016 6:27 AM, Raynaud Alexandre wrote:
> clamd check databases every hour and it takes 25s

I know Cryptowall is nasty but the problem isn't how long Clam takes to
update and for ASSP to recognize it is back up. The problem is you are
updating way too often.

- Bob


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Assp-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/assp-user