Installing and Configuring Snowflake agents (4369841)

Installing and Configuring Snowflake agents

How should Snowflake agents be configured in Foglight?

Installation instructions for Foglight for Snowflake are detailed in the following sections and should be performed in order:

• Snowflake Pre-Configuration
• Cartridge Installation
• Creating and Configuring Agents

Foglight for Snowflake Pre-Configuration

Monitoring requires the agent to be provided with a Snowflake user. That user should have sufficient privileges to access limited data and functions in system tables located in the Snowflake and user-created databases. It is recommended that you create a role and assign it to the user for this purpose. The following is a brief list of steps on how to do this. Skip any steps in configuration that were previously completed. For additional information, refer to the complete Snowflake User Management documentation.

Snowflake Role Setup

1. In the Snowflake console, with a user that has the ACCOUNTADMIN role selected, navigate to Admin>Users & Roles in the navigation pane on the left.
2. Select “Roles” at the top of the page, then “Table” below it and at the top-right of the page, click the “+ Role” button to create a new role and name it MONITORADMIN or a name of your choice.
3. Select the new role in the list of roles and then on the details page click the “…” at the top-right, then “Manage global privileges” in the menu.
4. In the Manage Global Privileges for Role popup, use the “Global privilege to grant” dropdown to select “Monitor Execution” and “Monitor Usage”, then click “Update Privileges”.
5. If the user already exists, click the “Grant to User” button in the bottommost section of the details page to grant the MONITORADMIN role to the user. Otherwise, the role can be attached during user creation.
6. Next, we need to assign the role the ability to execute queries on a warehouse. In the left navigation panel, go to Admin>Warehouses and select or create the warehouse that will be assigned to the monitoring user. In the bottommost section of the details page, grant the “USAGE” privilege to MONITORADMIN. Read Selecting a Compute Warehouse below for important information to consider when designating a warehouse.
7. Next, we need to assign the role privileges on data objects. In the left navigation panel, go to Data>Databases, and select the Snowflake database first. In the Privileges section, click the “+ Privilege” button and assign to the MONITORADMIN role “IMPORTED PRIVILEGES”, then click “Grant Privileges”.
8. For each other database that you wish for the agent to monitor, assign the USAGE role for the database to MONITORADMIN and for each existing pipe (found in database>schema>Pipes, the MONITOR privilege. You can also assign in the database “MONITOR - FUTURE PIPE” so that each new pipe that is created will automatically assign its monitoring privilege to MONITORADMIN.

Snowflake User Setup

1. In the Snowflake console, with a user that has the ACCOUNTADMIN role selected, navigate to Admin>Users & Roles in the navigation pane on the left.
2. At the top-right of the page, click the “+ User” button to create a new user and provide the relevant details.
3. Then, expand “Advanced User Options” and assign the default role MONITORADMIN or the role you have created with appropriate privileges and a default warehouse for the user to use when performing queries. Read the Selecting a Compute Warehouse section below for important information to consider when designating a warehouse.
4. Finish by clicking “Create User”.

FMS Configuration

Because data gathered from some Snowflake sources are only available after a window of latency, some additional configuration steps are required for the FMS to handle this delay correctly.

FMS Host Machine

1. Navigate to C:\Quest\Foglight\config or the corresponding directory on the host where your FMS is installed and open the server.config file.
2. In the section with server VM options, add the following line (increment the option number if other options are already present). This increases the time limit for historical data to be submitted from 2 hours to 3 hours.

   server.vm.option0 = "-Dfoglight.data_service.max_past_timestamp_delta=10800000"

3. Restart the FMS

FMS Web Console

1. In the FMS web console, navigate to Administration>Schedules>Manage Schedules in the left navigation tab.
2. Click on the “Daily Database Maintenance” schedule and then the item in the table and modify the start time to 3am. This ensures that any historical data collected for the previous day with a latency of up to 3 hours will be included in the nightly observation data rollup.

Selecting a Compute Warehouse

While the Snowflake agent requires minimal resources to execute monitoring queries, there are important technical and billing implications to consider when designating a warehouse for the Snowflake user to employ. The warehouse is assigned to the user either through the default warehouse setting for the Snowflake user or through the Agent Properties.

First, the Snowflake user must have the privilege to execute queries on a warehouse. This requires the USAGE privilege. It is preferred that the warehouse be either constantly running or set to auto-resume. Otherwise, an admin with OPERATE privilege on the warehouse will be required to start it from a suspended state. If auto-suspend is enabled, the warehouse will suspend itself when it has not received a query in the last x minutes. If the warehouse is set to auto-resume, then it will automatically resume to execute a query when one is submitted. There is generally a delay of several seconds in query execution when a warehouse resumes while it provisions resources, but this will have no impact on the agent’s data collection.

Secondly, it is important to understand how Snowflake charges for compute resources. Based on the size of a warehouse, Snowflake will charge x credits per hour of usage. While charges are calculated per second (minimum 60 seconds) of usage, usage simply means that the warehouse is in an active (not suspended or provisioning) state. This also means that if a warehouse were to resume from a suspended state to run queries for 5 seconds, it will still be charged for 60 seconds of usage. If it were to do this once a minute, the cost would be the same as if the warehouse was constantly active and running queries.

Therefore, there are two recommended configurations for maximum benefit at low cost:

1. Assign the monitoring user to a warehouse handling other workloads that has compute resources to spare that is either constantly running or not expected to receive a significant spike in query submissions while auto-suspended. Running agent queries on this type of warehouse will not result in any additional costs and very low impact on existing workload.
2. Run the agent on a dedicated warehouse. This will have no impact on other workloads, but will increase costs. However, we have several recommendations on how to reduce these costs:
   1. Use a warehouse of X-Small size. Agent queries do not require many compute resources.
   2. Enable auto-resume and auto-suspend with a very low suspension delay, like 15 seconds. Queuing while waiting for a suspended warehouse to resume will not affect data integrity.
   3. Set collection intervals in the Agent Properties to similar intervals and none shorter than 5 or 10 minutes. If different collections are run at the same time and at a longer interval, this will maximize the amount of time that a warehouse can remain suspended and thus not using credits.

Cartridge Installation

1. Open the Foglight Web Console.
2. From the navigation pane, select: Administration > Setup > Manage Licenses. The Licenses screen appears. Click the Install button in the table and then upload a valid Snowflake .license file from your computer.
3. From the navigation pane, select: Administration > Cartridges > Cartridge Inventory. The Cartridge Inventory screen appears.
4. Load the SnowflakeAgent-x_xx.car file by browsing to the location where the .car file exists and then clicking on "Install Cartridge." Leave the "Enable on Install" check box checked.
5. Once the installation is complete on the Foglight Management Server, the SnowflakeAgent Cartridge will appear in this list below as an installed cartridge.

For more information on these dashboards, read the Foglight User Guide.

Creating and Configuring Agents

Using the Agent Status Dashboard

The Agent Status page can be used to create new agents and configure and manage existing agents. To access the page from the navigation pane, select: Dashboards > Administration > Agents > Agent Status.

Use the following steps to create a new agent instance:

1. If the Snowflake agent package was not previously deployed to the FglAM hosting the agent, it must be done before an agent has been created, or it will be performed automatically when the first agent is created on that FglAM. You can use the Deploy Agent Package button on the Agent Status or Agent Managers page to perform this.
2. Click the Create Agent button and follow the instructions for the cards:
   1. Host Selector - Choose the Agent Manager on which the agent should run. Considerations for this may include physical or virtual locality to the monitored instance, allocated resources, or grouping with other agents of the same type or monitored environment.
   2. Agent Type and Instance Name – Select the SnowflakeAgent type. Then, select the Specify Name radio button and provide a name for the agent that will be created. This is not canonical and should be representative of the account that this agent will monitor.
   3. Summary – Click Finish.
3. Once the agent has been created, click the checkbox next to the Snowflake agent.
4. Click the Edit Properties button.
5. Select Modify the default properties for this agent.
6. Edit the agent properties for the Snowflake agent instance as described in Knowledge base article 4369842
7. Click the Activate button.

To modify the properties for an existing agent, skip to step 3 and Deactivate, then Reactivate the agents after changing the configuration.

 

This article is current as of the 6.3.0 release of the Snowflake cartridge. Consult the Foglight for Snowflake user guide for the most current information.