Re: [Gluster-devel] RFC for quota limit exceeded logging

[Anand Babu Periasamy]

+1 on Vijay's comment. Logging infrastructure is useless for quota monitoring. It should be handled via user interface (CLI and alerting). gluster cli can report them similar to volume top and profile commands. Even if we dont implement alert, sysadmin may simply put a cron job around gluster cli.

On Mon, Nov 25, 2013 at 8:57 AM, Vijay Bellur <address@hidden> wrote:

On 11/19/2013 03:08 PM, Anuradha Talur wrote:

Hi all,

This mail is regarding quota logging for the following conditions :

1. Soft-limit alerts - "Usage crossed soft-limit" and "usage above soft-limit"
2. Quota exceeded warning

Issue with the soft-limit alerts logging --
Currently, the soft-limit alerts are logged in the brick logs when a write
is made and soft-limit is exceeded. In a scenario where there are a large
number of bricks, it is difficult for the system admin to poll all the
bricks for a soft-limit alert.

If polling is the mechanism for a soft-limit alert, why not have it as part of a command? The quota list CLI or an alternate new command can provide the list of directories on which soft-limit has been breached. This CLI can be polled to determine which of the directories exceeds soft-limit.

Issue with the quota exceeded logging --
When the quota exceeds, op_errno is set to EDQUOT and eventually
"Quota exceeded" logs are being logged in the client logs (nfs.log, fuse mnt log,..)
as of now (Similar to other op_errors logging).
An issue has been raised stating that the quota hard-limit and soft-limit
exceeding logs should be logged in same file.

Why is logging being preferred over CLI? If logging has to be the preferred mechanism for alerting, can we not use LOG_ALERT across all log files?

Logs being seen in nfs.log and fuse mount log are useful in determining what errors are being returned to applications. In addition logging in fuse and bricks can potentially be used by different consumers. The fuse log files would be useful for an administrator who manages compute and the brick log files are useful for an administrator who manages storage.

So a need for a unified logging has been proposed. We came up with the
solution that all the soft-limit alert logs and the quota exceeded logs
should be logged in quotad.log file.

Why cannot these logs be in the brick log files? Since both quota and marker xlators are part of the brick stack, logging in brick log files seems to be a better choice.

Resulting in the limit exceeded logs being logged in all the nodes in the
cluster. The sys admin can poll any one node to get the status of soft-limit
and hard-limit.

Wouldn't we be flooding all nodes with similar information? With a centralized logging infrastructure, we might end up with a lot of repeated/redundant logs.

Approach proposed to achieve this solution --
During the event of soft-limit/hard-limit exceeding, glusterd is informed.
glusterd of the current node should interact with the glusterd of the other
nodes in the cluster indicating the issue.
All the glusterd's talk to their respective quotad to log the limit exceed
in their logs.

The following bugs are related to the points made here --
https://bugzilla.redhat.com/show_bug.cgi?id=1020816
https://bugzilla.redhat.com/show_bug.cgi?id=1019302


Is this approach recommended?

I don't think this is a very clean approach. Having a CLI and/or providing the right log messages in brick log files at level LOG_ALERT seems to be good enough for alerting.

-Vijay

_______________________________________________
Gluster-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/gluster-devel

--
-ab

Imagination is more important than knowledge --Albert Einstein