VictoriaLogs + Alloy Binary Deployment

Prerequisites

If you have not rotated MongoDB logs yet, first read MongoDB Log Rotation. This option does not include Grafana, so a usable Grafana environment must already be available. When installing the VictoriaLogs datasource plugin, Grafana must meet the plugin requirements. The current reference VictoriaLogs datasource plugin version is 0.32.0, Grafana must be >= 10.4.0. If this requirement is not met, first refer to Grafana Upgrade Steps Reference.

Deploy VictoriaLogs

1. Prepare the installation directory

   mkdir -p /data/mdtemp
   mkdir -p /usr/local/victorialogs
   mkdir -p /data/victorialogs
   cd /data/mdtemp

2. Download the VictoriaLogs package

   wget https://pdpublic.mingdao.com/private-deployment/offline/common/victoria-logs-linux-amd64-v1.52.0.tar.gz

3. Extract it to the installation directory

   tar xzvf victoria-logs-linux-amd64-v1.52.0.tar.gz -C /usr/local/victorialogs/
   chmod +x /usr/local/victorialogs/victoria-logs-prod

   Verify the version:

   /usr/local/victorialogs/victoria-logs-prod --version

   The expected version is v1.52.0.

4. Create the data directory

   mkdir -p /data/victorialogs

5. Write the environment file

   cat > /usr/local/victorialogs/victorialogs.env <<'EOF'
   # Data directory
   STORAGE_DATA_PATH=/data/victorialogs
   
   # HTTP listen port
   HTTP_LISTEN_ADDR=:9428
   
   # Basic Auth; change this in production
   HTTP_AUTH_USERNAME=myuser
   HTTP_AUTH_PASSWORD=mypassword
   
   # Data retention
   RETENTION_PERIOD=30d
   RETENTION_DISK_USAGE=50GiB
   
   # Memory and concurrency
   MEMORY_ALLOWED_BYTES=1GiB
   MAX_CONCURRENT_INSERTS=4
   MAX_CONCURRENT_REQUESTS=2
   
   # Query limits
   SEARCH_MAX_QUERY_DURATION=30s
   SEARCH_MAX_QUERY_TIMERANGE=24h
   SEARCH_MAX_QUEUE_DURATION=5s
   SEARCH_LOG_SLOW_QUERY_DURATION=3s
   
   # Write protection
   INSERT_MAX_LINE_SIZE_BYTES=262144
   INSERT_MAX_FIELDS_PER_LINE=300
   
   # Log level
   LOGGER_LEVEL=INFO
   EOF

   The following parameters must be changed for the target environment:

   HTTP_AUTH_USERNAME
   HTTP_AUTH_PASSWORD
   RETENTION_PERIOD
   RETENTION_DISK_USAGE

6. Write the systemd service file

   cat > /etc/systemd/system/victorialogs.service <<'EOF'
   [Unit]
   Description=VictoriaLogs Logging Service
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   Type=simple
   EnvironmentFile=/usr/local/victorialogs/victorialogs.env
   Environment="GOMEMLIMIT=2GiB"
   
   WorkingDirectory=/usr/local/victorialogs
   
   ExecStart=/usr/local/victorialogs/victoria-logs-prod \
     -storageDataPath=${STORAGE_DATA_PATH} \
     -httpListenAddr=${HTTP_LISTEN_ADDR} \
     -retentionPeriod=${RETENTION_PERIOD} \
     -retention.maxDiskSpaceUsageBytes=${RETENTION_DISK_USAGE} \
     -memory.allowedBytes=${MEMORY_ALLOWED_BYTES} \
     -maxConcurrentInserts=${MAX_CONCURRENT_INSERTS} \
     -search.maxConcurrentRequests=${MAX_CONCURRENT_REQUESTS} \
     -search.maxQueryDuration=${SEARCH_MAX_QUERY_DURATION} \
     -search.maxQueryTimeRange=${SEARCH_MAX_QUERY_TIMERANGE} \
     -search.maxQueueDuration=${SEARCH_MAX_QUEUE_DURATION} \
     -search.logSlowQueryDuration=${SEARCH_LOG_SLOW_QUERY_DURATION} \
     -insert.maxLineSizeBytes=${INSERT_MAX_LINE_SIZE_BYTES} \
     -insert.maxFieldsPerLine=${INSERT_MAX_FIELDS_PER_LINE} \
     -httpAuth.username=${HTTP_AUTH_USERNAME} \
     -httpAuth.password=${HTTP_AUTH_PASSWORD} \
     -loggerLevel=${LOGGER_LEVEL}
   
   Restart=on-failure
   RestartSec=3s
   
   LimitNOFILE=65536
   TasksMax=2048
   
   CPUAccounting=true
   CPUQuota=100%
   
   MemoryAccounting=true
   MemoryMax=2G
   MemorySwapMax=0
   
   StandardOutput=journal
   StandardError=journal
   
   [Install]
   WantedBy=multi-user.target
   EOF

7. Start VictoriaLogs

   systemctl daemon-reload
   systemctl enable --now victorialogs
   systemctl status victorialogs --no-pager
   ss -lnt | grep '9428'
   curl -s -u 'myuser:mypassword' http://localhost:9428/health
   journalctl -u victorialogs -n 100 --no-pager

Deploy Alloy

Deploy Alloy on every MongoDB node. It reads the local mongodb.log and pushes slow-query logs to VictoriaLogs. Deploy one Alloy instance per node to collect its local logs, so that each slow-query log can be associated with its source node.

Points to note:

• Change VICTORIALOGS_PUSH_URL to the actual VictoriaLogs address.
• Change VICTORIALOGS_USERNAME and VICTORIALOGS_PASSWORD to the actual authentication credentials.
• Alloy automatically uses the local IPv4 address as the job identifier; no manual node-name change is needed.
• Set MONGODB_LOG_PATH to the absolute path of the MongoDB log file on the current node.
• /data/alloy-vlogs-data stores the log read position. Do not delete it arbitrarily, or logs may be collected again.

1. Confirm the MongoDB log path

   ls -lh /data/logs/mongodb/mongodb.log
   tail -n 3 /data/logs/mongodb/mongodb.log

2. Create the Alloy installation directory

   mkdir -p /data/mdtemp
   mkdir -p /usr/local/alloy-vlogs/bin
   mkdir -p /data/alloy-vlogs-data
   cd /data/mdtemp

3. Download the Alloy package

   wget https://pdpublic.mingdao.com/private-deployment/offline/common/alloy-linux-amd64-v1.19.2.zip

4. Extract and install Alloy

   unzip -o alloy-linux-amd64-v1.19.2.zip -d /data/mdtemp/
   mv -f /data/mdtemp/alloy-linux-amd64 /usr/local/alloy-vlogs/bin/alloy

   chmod +x /usr/local/alloy-vlogs/bin/alloy
   /usr/local/alloy-vlogs/bin/alloy --version

   The expected version is v1.19.2.

5. Write the Alloy environment file

   cat > /usr/local/alloy-vlogs/alloy.env <<'EOF'
   # VictoriaLogs address; change this to the actual IP
   VICTORIALOGS_PUSH_URL="http://VictoriaLogs_IP:9428/insert/loki/api/v1/push?preserve_json_keys=attr&_stream_fields=job,host&ignore_fields=filename"
   
   # VictoriaLogs Basic Auth
   VICTORIALOGS_USERNAME="myuser"
   VICTORIALOGS_PASSWORD="mypassword"
   
   # Absolute path to the MongoDB log
   MONGODB_LOG_PATH="/data/logs/mongodb/mongodb.log"
   EOF
   chmod 600 /usr/local/alloy-vlogs/alloy.env

   The following parameters must be changed for the target environment:

   VictoriaLogs_IP
   VICTORIALOGS_USERNAME
   VICTORIALOGS_PASSWORD
   MONGODB_LOG_PATH

6. Write the Alloy configuration file

   cat > /usr/local/alloy-vlogs/config.alloy <<'EOF'
   // ============================================================
   // Grafana Alloy
   // MongoDB Slow Query -> VictoriaLogs
   // ============================================================
   
   logging {
     level  = "info"
     format = "logfmt"
   }
   
   // ============================================================
   // 1. Read MongoDB logs
   // ============================================================
   loki.source.file "mongodb" {
     targets = [
       {
         __path__ = sys.env("MONGODB_LOG_PATH"),
         job      = string.format("mongodb-%s", sys.env("HOST_IP")),
         host     = constants.hostname,
       },
     ]
   
     forward_to = [
       loki.process.mongodb.receiver,
     ]
   }
   
   // ============================================================
   // 2. Process MongoDB slow queries
   // ============================================================
   loki.process "mongodb" {
     forward_to = [
       loki.write.victorialogs.receiver,
     ]
   
     // Parse MongoDB JSON logs
     stage.json {
       expressions = {
         ts             = "t.\"$date\"",
         severity       = "s",
         c              = "",
         msg            = "",
         attr           = "attr",
         durationMillis = "attr.durationMillis",
         ns             = "attr.ns",
         planSummary    = "attr.planSummary",
         docsExamined   = "attr.docsExamined",
         reslen         = "attr.reslen",
       }
     }
   
     // Use the original MongoDB log timestamp
     stage.timestamp {
       source = "ts"
       format = "RFC3339"
     }
   
     // c / msg are used only for the filters below
     stage.labels {
       values = {
         c   = "",
         msg = "",
       }
     }
   
     // Keep only c=COMMAND
     stage.match {
       selector            = "{c!=\"COMMAND\"}"
       action              = "drop"
       drop_counter_reason = "not_command"
     }
   
     // Keep only msg=Slow query
     stage.match {
       selector            = "{msg!=\"Slow query\"}"
       action              = "drop"
       drop_counter_reason = "not_slow_query"
     }
   
     // Remove temporary labels
     stage.label_drop {
       values = [
         "c",
         "msg",
       ]
     }
   
     // Convert the MongoDB log level
     stage.template {
       source = "level"
   
       template = `{{ if eq .severity "I" }}info{{ else if eq .severity "W" }}warn{{ else if eq .severity "E" }}error{{ else if eq .severity "F" }}critical{{ else if eq .severity "D" }}debug{{ else }}info{{ end }}`
     }
   
     stage.labels {
       values = {
         level = "",
       }
     }
   
     // Keep common fields as Structured Metadata
     stage.structured_metadata {
       values = {
         durationMillis = "",
         ns             = "",
         planSummary    = "",
         docsExamined   = "",
         reslen         = "",
       }
     }
   
     // Content displayed in the Grafana Logs panel
     stage.template {
       source = "display_msg"
   
       template = `{
     "ns": {{ toJson .ns }},
     "durationMillis": {{ toJson (printf "%v" .durationMillis) }},
     "planSummary": {{ toJson .planSummary }},
     "docsExamined": {{ toJson (printf "%v" .docsExamined) }},
     "reslen": {{ toJson (printf "%v" .reslen) }},
     "attr": {{ toJson (fromJson .attr) }}
   }`
     }
   
     // Final JSON sent to VictoriaLogs
     stage.template {
       source = "vl_line"
   
       template = `{"_msg":{{ toJson .display_msg }},"ns":{{ toJson .ns }},"durationMillis":{{ toJson (printf "%v" .durationMillis) }},"planSummary":{{ toJson .planSummary }},"docsExamined":{{ toJson (printf "%v" .docsExamined) }},"reslen":{{ toJson (printf "%v" .reslen) }},"attr":{{ toJson (fromJson .attr) }}}`
     }
   
     stage.output {
       source = "vl_line"
     }
   }
   
   // ============================================================
   // 3. Send to VictoriaLogs
   // ============================================================
   loki.write "victorialogs" {
     endpoint {
       url = sys.env("VICTORIALOGS_PUSH_URL")
   
       batch_wait = "1s"
       batch_size = "100KiB"
   
       basic_auth {
         username = sys.env("VICTORIALOGS_USERNAME")
         password = sys.env("VICTORIALOGS_PASSWORD")
       }
     }
   }
   EOF

7. Validate the Alloy configuration

   Load the environment variables:

   set -a
   . /usr/local/alloy-vlogs/alloy.env
   set +a

   Temporarily generate the local IP:

   export HOST_IP="$(ip -4 route get 1.1.1.1 2>/dev/null \
     | sed -n 's/.* src \([^ ]*\).*/\1/p' \
     | head -n1)"

   Confirm the environment variables:

   echo "$HOST_IP"
   echo "$MONGODB_LOG_PATH"

   Validate the configuration:

   /usr/local/alloy-vlogs/bin/alloy validate \
     /usr/local/alloy-vlogs/config.alloy

   Continue only if no errors are reported.

8. Write the Alloy startup script

   cat > /usr/local/alloy-vlogs/bin/start-alloy.sh <<'EOF'
   #!/bin/sh
   
   # Automatically obtain the server egress IP from the default IPv4 route
   HOST_IP=$(ip -4 route get 1.1.1.1 2>/dev/null \
     | sed -n 's/.* src \([^ ]*\).*/\1/p' \
     | head -n1)
   
   if [ -z "$HOST_IP" ]; then
       echo "ERROR: failed to detect local IPv4 address"
       exit 1
   fi
   
   if [ -z "$MONGODB_LOG_PATH" ]; then
       echo "ERROR: MONGODB_LOG_PATH is empty"
       exit 1
   fi
   
   if [ ! -r "$MONGODB_LOG_PATH" ]; then
       echo "ERROR: MongoDB log is not readable: $MONGODB_LOG_PATH"
       exit 1
   fi
   
   export HOST_IP
   
   echo "Alloy HOST_IP=${HOST_IP}"
   echo "MongoDB LOG=${MONGODB_LOG_PATH}"
   
   /usr/local/alloy-vlogs/bin/alloy validate \
     /usr/local/alloy-vlogs/config.alloy || exit 1
   
   exec /usr/local/alloy-vlogs/bin/alloy run \
     --server.http.listen-addr=127.0.0.1:9081 \
     --storage.path=/data/alloy-vlogs-data \
   /usr/local/alloy-vlogs/config.alloy
   EOF

   Grant execute permission:

   chmod +x /usr/local/alloy-vlogs/bin/start-alloy.sh

   Verify the automatically detected IP:

   ip -4 route get 1.1.1.1

   View only the IP:

   ip -4 route get 1.1.1.1 \
     | sed -n 's/.* src \([^ ]*\).*/\1/p'

9. Write the systemd service file

   cat > /etc/systemd/system/alloy-vlogs.service <<'EOF'
   [Unit]
   Description=Grafana Alloy VictoriaLogs Collector
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   Type=simple
   
   EnvironmentFile=/usr/local/alloy-vlogs/alloy.env
   
   Environment="GOMEMLIMIT=768MiB"
   
   ExecStart=/usr/local/alloy-vlogs/bin/start-alloy.sh
   
   Restart=on-failure
   RestartSec=3s
   
   CPUAccounting=true
   CPUQuota=100%
   
   MemoryAccounting=true
   MemoryMax=1G
   MemorySwapMax=0
   
   LimitNOFILE=65536
   TasksMax=1024
   
   StandardOutput=journal
   StandardError=journal
   
   [Install]
   WantedBy=multi-user.target
   EOF

10. Start Alloy

systemctl daemon-reload
systemctl enable --now alloy-vlogs

View the status:

systemctl status alloy-vlogs --no-pager

View the logs:

journalctl -u alloy-vlogs -n 50 --no-pager

The output should include:

Alloy HOST_IP=<local_IP>
MongoDB LOG=/data/logs/mongodb/mongodb.log

11. Alloy health checks

Port:

ss -lntp | grep ':9081'

Ready：

curl -s http://127.0.0.1:9081/-/ready

Expected response:

Alloy is ready.

Healthy：

curl -s http://127.0.0.1:9081/-/healthy

Expected response:

All Alloy components are healthy.

12. Check Alloy collection and delivery

curl -s http://127.0.0.1:9081/-/healthy
journalctl -u alloy-vlogs -n 50 --no-pager
curl -s http://127.0.0.1:9081/metrics | egrep -i 'loki_.*(sent|dropped|error|retry|read)' | head -n 80

Configure Grafana

This option does not include Grafana, so a usable Grafana environment must already be available. If Grafana is not available, install or upgrade Grafana before continuing with the steps below. The current reference VictoriaLogs datasource plugin version is 0.32.0. Grafana must be >= 10.4.0 during installation. If this requirement is not met, first refer to Grafana Upgrade Steps Reference.

1. Install the VictoriaLogs datasource plugin

   cd /data/mdtemp
   # Grafana >= 10.4.0; upgrade Grafana first if this requirement is not met
   wget https://pdpublic.mingdao.com/private-deployment/offline/common/victoriametrics-logs-datasource-v0.32.0.tar.gz
   tar xzvf victoriametrics-logs-datasource-v0.32.0.tar.gz -C /usr/local/grafana/data/plugins/
   systemctl restart grafana

2. Configure the datasource

   • Enter the VictoriaLogs address in url, for example http://127.0.0.1:9428.
   • Enable Basic auth.
   • Enter HTTP_AUTH_USERNAME in User.
   • Enter HTTP_AUTH_PASSWORD in Password.
   • Keep Maximum lines at 2000000 unless the environment requires otherwise.

3. Import the slow-query dashboard

   The dashboard file is ready. Download it and import it into Grafana.

   View / download the dashboard JSON

   • Dashboard file: mongodb-VictoriaLogs-Slow-Query-v1.1.0.en.json
   • Import method: in Grafana, click Import dashboard and upload this file.
   • After importing, confirm that the current VictoriaLogs datasource is selected.
   • To preview the content, open this JSON file directly in a browser.

   Download the dashboard JSON
   Open the JSON file in a browser