Description
Anomaly detect is a new CLI option for system CLI, storage_group CLI, cloud_tier CLI, and performance_tier CLI to detect anomalies system-wide, storage-group level, or container level.
Syntax
system --anomaly_detect --set <ON|OFF> --interval <30 to 4320> --metric <os-auth | ui-auth | proto-auth | all>
Where
set - Specified to turn on/off particular metric for anomaly detection interval - Specifies(in minutes) how frequently anomaly detection has to run. Valid values are 30 to 4320(i.e. 3-days) os-auth - OS or system level authentication failures/anomalies
ui-auth - UI user authentication failures/anomalies
proto-auth - Protocol authentications mainly RDS/OST
storage_group CLI
storage_group --anomaly_detect --name <group name>
--set <ON | OFF>
--metric <ingest | savings | expiry | all>
cloud_tier CLI
cloud_tier --anomaly_detect [--name <cloud tier name>]
--set <ON | OFF>
--metric <ingest | savings | expiry | all>
[--cloud_archive]
performance-tier CLI:
performance_tier --anomaly_detect --set <ON | OFF>
--metric <ingest | savings | expiry | all>
metrics used in above CLI commands:
ingest - Anomalies in ingested bytes or backup pattern, applicable to containers under the storage group.
savings - Anomalies in post-dedupe and post-compress bytes only storage group.
expiry - only applicable to containers in the storage group.
all - above all metrics
Container CLI
container --anomaly_detect --name <container name>
--set <ON | OFF | INHERIT>
--metric <ingest | expiry | all>
Where
INHERIT - Specifies to use same metric set on storage group. This is default when container is created. metric: ingest - Anomalies in ingested bytes and overwrite bytes.
expiry - Anomalies in files unlinked and bytes unlinked.
Viewing anomaly settings
Anomaly detection settings can be viewed at system level or storage group level or cloud_tier or performance_tier or container level using existing 'show' options using the following commands.
System CLI
system --show ... ... Anomaly Detection : Enabled Anomaly Detection Interval(min) : 30 Anomaly Detection Metric : os-auth,ui-auth,proto-auth ... Anomaly Detection Service : RUNNING Mar 22 03:11:59
(displaying only new fields in the output)
Storage group CLI
storage_group --show --name <group name>
Storage_group Name : DefaultGroup
Storage_group Compression Type : Fast
Storage_group Encryption Set : Off
Storage_group Encryption Type : Off
Storage_group Rotate Period : 0
Storage_group Passphrase set : No
Storage_group Quota limit : Unlimited
Storage_group Anomaly Detection : Enabled
Storage_group Anomaly Detection Interval(min): 30
Storage_group Anomaly Metric : ingest,savings,expiry
DefaultGroup's Containers
-------------------------
rdas (id: 1)
Container CLI
container --show --name <name>
Container's Group ID : 0
Container's Group Name : DefaultGroup
Container Name : rdas
Container Path : /containers/rdas
Container Object Direct Storage : No
Container Cloud Replication : None
Container cloud locking : None
Container Marker : None
Recycle Bin Enabled : No
RDS connection Quota : Unlimited
RDS connection Used Capacity : 0.0 GiB
RDS connection Enabled : Yes
RDS connection status : Available
Number of container UACs : 1
Anomaly detection : Enabled
Anomaly detection interval(min) : 30
Anomaly metric : ingest,expiry [inherit]
Cloud_tier CLI
cloud_tier --show --name <cloud tier name>
Cloud_tier Name : minio-ct
Cloud_tier Compression Type : Fast
Cloud_tier Encryption Set : On
Cloud_tier Encryption Type : Static
Cloud_tier Rotate Period : 0
Cloud_tier Passphrase set : Yes
Cloud_tier Anomaly Detection : Disabled
Cloud_tier Anomaly Detection Interval(min): 30
Cloud_tier Anomaly Metric : ingest,savings,expiry
Performance_tier CLI
performance_tier --show
Storage_group Name : PerformanceTier
Storage_group Compression Type : Fast
Storage_group Encryption Set : On
Storage_group Encryption Type : Static
Storage_group Rotate Period : 30
Storage_group Passphrase set : Yes
Storage_group Mapped Enclosure : /perf_tier
Storage_group Quota limit : Unlimited
Storage_group Anomaly Detection : Enabled
Storage_group Anomaly Detection Interval(min): 30
Storage_group Anomaly Metric : ingest,savings,expiry
PerformanceTier's Containers
----------------------------
rds_perf (id: 112)
nfs_perf (id: 121)
Anomaly detection report
Following CLI can used to view anomalies detected so far.
Usage:
/opt/qorestor/bin/ocamlreport [--cont=<container_name> --group=<group_name> --start_date=<start date YYYY-MM-DD> --end_date=<end date YYYY-MM-DD> --start_time=<HH:MM:SS> --end_time=<HH:MM:SS> --skip_group --skip_cont --skip_allauth --skip_osauth --skip_uiauth --skip_protoauth --deleted] [ --latest]
To report all anomalies, execute without options
ocamlreport
To report anomalies only for authentications
ocamlreport --skip_group --skip_cont
To report anomalies only for one type of authentication (skip other auth)
ocamlreport --skip_group --skip_cont --skip_uiauth --skip_protoauth
To report anomalies only on container
ocamlreport --cont=<> --skip_allauth
To report anomalies only on storage group
ocamlreport --group=<> --skip_cont --skip_allauth
To report anomalies for all storage groups
ocamlreport --skip_cont --skip_allauth
To report anomalies for all containers in a storage group
ocamlreport --group=<> --skip_group --skip_allauth
To report anomalies on all deleted storage group and containers
ocamlreport --deleted --skip_allauth
To report anomalies in csv file (generates under diagnostics directory)
ocamlreport --csv
Generated anomaly report: /var/diagnostic_logs/oca_anomaly_report_<hostname>_2024-04-12_12-49-17.csv
Training/Retraining
You can manually train or retrain the anomaly detection on-demand . An automatic retraining occurs every 30-days.
Syntax
/opt/qorestor/bin/ocamltrain [--start_date=YYYY-MM-DD] [--sg_only=<sg-name> | --cont=<cont-name>] --train_days=<days 30 to 90> | --versions
|
|
NOTE: Difference between current date and start-date should accommodate/contain at-least train-days. |
To train all storage groups and containers
ocamltrain --start_date=YYYY-MM-DD --train_days=<days 30 to 90>
To train a particular storage group only
ocamltrain --start_date=YYYY-MM-DD --train_days=<days 30 to 90> --sg_only=sg-name
To train a particular container
ocamltrain --start_date=YYYY-MM-DD --train_days=<days 30 to 90> --cont=cont-name
To list all storage groups/container ML versions
ocamltrain --versions
To list storage group versions
ocamltrain --sg_only=<sg-name> --versions
To list container ML versions
ocamltrain --cont=<cont-name> --versions
|
|
NOTE: Training won't run on already [marked] deleted containers/storage groups. |
Configuring Email
To configure an email for the anomalies execute the following CLI.
/opt/qorestor/bin/email_anomalies --configure