VictoriaLogs + Promtail Binary Deployment

Prerequisites

If you have not rotated MongoDB logs yet, please first read MongoDB Log Rotation. This deployment option does not include Grafana, so a usable Grafana environment must already be available. When installing the VictoriaLogs datasource plugin, Grafana must meet the plugin requirements. The current reference VictoriaLogs datasource plugin version is 0.23.5, and Grafana must be >= 10.4.0. If your Grafana version does not meet this requirement, please first refer to Grafana 12.1.2 Upgrade Steps Reference.

Deploy VictoriaLogs

Prepare the installation directory
```
mkdir -p /data/mdtemp
cd /data/mdtemp
```

Download the VictoriaLogs package

wget https://pdpublic.mingdao.com/private-deployment/offline/common/victoria-logs-linux-amd64-v1.43.1.tar.gz

Extract it to the installation directory

mkdir /usr/local/victorialogs/
tar xzvf victoria-logs-linux-amd64-v1.43.1.tar.gz -C /usr/local/victorialogs/

Create the data directory
```
mkdir -p /data/victorialogs
```

Write the environment file

cat > /usr/local/victorialogs/victorialogs.env <<'EOF'
# VictoriaLogs basic configuration
STORAGE_DATA_PATH=/data/victorialogs
HTTP_LISTEN_ADDR=:9428
HTTP_AUTH_USERNAME=myuser
HTTP_AUTH_PASSWORD=mypassword

# Data retention policy
RETENTION_PERIOD=30d
RETENTION_DISK_USAGE=50GiB

# Memory and concurrency controls
MEMORY_ALLOWED_PERCENT=2
MAX_CONCURRENT_INSERTS=4
MAX_CONCURRENT_REQUESTS=2

# Query limits
SEARCH_MAX_QUERY_DURATION=30s
SEARCH_MAX_QUERY_TIMERANGE=24h
SEARCH_MAX_QUEUE_DURATION=5s
SEARCH_LOG_SLOW_QUERY_DURATION=3s

# Write protection
INSERT_MAX_LINE_SIZE_BYTES=262144
INSERT_MAX_FIELDS_PER_LINE=300

# syslog extra fields
SYSLOG_EXTRA_FIELDS_TCP={"job":"rsyslog"}

# Log level
LOGGER_LEVEL=INFO
EOF

Variable notes:

HTTP_AUTH_USERNAME and HTTP_AUTH_PASSWORD are used for VictoriaLogs Basic Auth. It is recommended to replace them with a more complex username and password according to your local standards instead of using the example values directly.
RETENTION_PERIOD is used to set the log retention period. For example, 30d means logs are retained for 30 days.
RETENTION_DISK_USAGE is used to set the disk usage upper limit for VictoriaLogs. For example, 50GiB means VictoriaLogs can use up to 50 GiB of disk space.
If the log volume is large, it is recommended to tune these two parameters together to prevent disk growth from getting out of control.

Write the systemd service file

cat > /etc/systemd/system/victorialogs.service <<'EOF'
[Unit]
Description=VictoriaLogs Logging Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
EnvironmentFile=/usr/local/victorialogs/victorialogs.env
Environment="GOMEMLIMIT=2GiB"
WorkingDirectory=/usr/local/victorialogs
ExecStart=/usr/local/victorialogs/victoria-logs-prod \
  -storageDataPath=${STORAGE_DATA_PATH} \
  -httpListenAddr=${HTTP_LISTEN_ADDR} \
  -retentionPeriod=${RETENTION_PERIOD} \
  -retention.maxDiskSpaceUsageBytes=${RETENTION_DISK_USAGE} \
  -memory.allowedPercent=${MEMORY_ALLOWED_PERCENT} \
  -maxConcurrentInserts=${MAX_CONCURRENT_INSERTS} \
  -search.maxConcurrentRequests=${MAX_CONCURRENT_REQUESTS} \
  -search.maxQueryDuration=${SEARCH_MAX_QUERY_DURATION} \
  -search.maxQueryTimeRange=${SEARCH_MAX_QUERY_TIMERANGE} \
  -search.maxQueueDuration=${SEARCH_MAX_QUEUE_DURATION} \
  -search.logSlowQueryDuration=${SEARCH_LOG_SLOW_QUERY_DURATION} \
  -insert.maxLineSizeBytes=${INSERT_MAX_LINE_SIZE_BYTES} \
  -insert.maxFieldsPerLine=${INSERT_MAX_FIELDS_PER_LINE} \
  -syslog.extraFields.tcp=${SYSLOG_EXTRA_FIELDS_TCP} \
  -httpAuth.username=${HTTP_AUTH_USERNAME} \
  -httpAuth.password=${HTTP_AUTH_PASSWORD} \
  -loggerLevel=${LOGGER_LEVEL}
ExecStop=/usr/bin/kill $MAINPID
Restart=on-failure
LimitNOFILE=65536
TasksMax=2048
CPUAccounting=true
CPUQuota=100%
MemoryAccounting=true
MemoryMax=2G
MemoryLimit=2G
MemorySwapMax=0
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

Start VictoriaLogs

systemctl daemon-reload
systemctl enable --now victorialogs
systemctl status victorialogs --no-pager
ss -lnt | grep '9428'
curl -s http://localhost:9428/health

Deploy Promtail

Promtail must be deployed on every MongoDB node. It reads the local mongodb.log file directly and pushes the slow query logs to VictoriaLogs. Each node needs its own Promtail instance so that the log source can be mapped back to the node one by one.

Notes:

PROMTAIL_PUSH_URL must be changed to the actual VictoriaLogs address.
PROMTAIL_USERNAME and PROMTAIL_PASSWORD must be changed to the real VictoriaLogs authentication credentials.
PROMTAIL_JOB_NAME must remain unique on each MongoDB node so that log sources can be distinguished.
PROMTAIL_LOG_PATH must be changed to the absolute path of the MongoDB log file on the current node.
If the log path differs between MongoDB nodes, adjust the corresponding configuration on each node separately.

Download Promtail

wget https://pdpublic.mingdao.com/private-deployment/offline/common/promtail-linux-amd64.zip

Extract and install

mkdir /usr/local/promtail-vlogs
unzip promtail-linux-amd64.zip -d /usr/local/promtail-vlogs/

Write the configuration file

cat > /usr/local/promtail-vlogs/promtail-config.yaml << 'EOF'
server:
  # Listening port
  http_listen_port: 9081
  # gRPC server listening port; 0 means random
  grpc_listen_port: 0
  # gRPC maximum receive message size, default 4 MB
  grpc_server_max_recv_msg_size: 4194304
  # gRPC maximum send message size, default 4 MB
  grpc_server_max_send_msg_size: 4194304
positions:
  # File used by Promtail to persist read offsets so collection can continue after restart
  filename: /usr/local/promtail-vlogs/positions.yaml
clients:
  - url: "http://10.206.0.5:9428/insert/loki/api/v1/push" # Change this to the VictoriaLogs address
    basic_auth:
      username: "myuser"  # Change this to the VictoriaLogs username
      password: "mypassword"  # Change this to the VictoriaLogs password
    batchwait: 1s        # Send a batch at most once per second
    batchsize: 102400    # About 100 KB
    headers:
      X-Loki-Disable-Structured-Metadata: "true"
scrape_configs:
  - job_name: mongodb
    static_configs:
      - targets:
          - localhost
        labels:
          job: mongodb-01  # Do not reuse this value on every node
          host: ${HOSTNAME}
          __path__: /data/logs/mongodb/mongodb.log # Absolute path to mongodb.log
    pipeline_stages:
      # 1. Parse JSON
      - json:
          expressions:
            ts: t."$date"
            c: c
            msg: msg
            attr: attr
      # 2. Use the timestamp from the log itself
      - timestamp:
          source: ts
          format: RFC3339
      # Add labels for later match stages
      - labels:
          c:
          msg:
      # 3. Keep only COMMAND and msg=Slow query logs. During initial deployment, you can comment out these two drop rules for validation.
      - match:
          selector: '{c!="COMMAND"}'
          action: drop
      - match:
          selector: '{msg!="Slow query"}'
          action: drop
      # 4. Promote fields under attr to the top level for easier LogSQL filtering
      - json:
          source: attr
          expressions:
            durationMillis: durationMillis
            ns: ns
            planSummary: planSummary
            docsExamined: docsExamined
            reslen: reslen
      # 5. Add labels for page display and LogSQL filtering
      - labels:
          durationMillis:
          ns:
          planSummary:
          docsExamined:
          reslen:
          ts:
          attr:
      # 6. Define _msg
      - template:
          source: line
          destination: line
          template: >-
            - {
              "ns":"{{ .ns }}",
              "durationMillis":"{{ .durationMillis }}",
              "planSummary":"{{ .planSummary }}",
              "docsExamined":"{{ .docsExamined }}",
              "reslen":"{{ .reslen }}",
              "attr":"{{ .attr }}"
            }
      - output:
          source: line
limits_config:
  readline_rate: 1000000  # Read rate limit: 1 MB/s
  readline_burst: 2000000  # Burst read limit: 2 MB
EOF

Write the positions file

cat > /usr/local/promtail-vlogs/positions.yaml << 'EOF'
positions: {}
EOF

Write the systemd service file

cat > /etc/systemd/system/promtail-vlogs.service << 'EOF'
[Unit]
Description=promtail Logging Service
After=network.target

[Service]
Type=simple
# Upper limit for the Go runtime heap; still effective even if cgroup detection fails
Environment="GOMEMLIMIT=1GiB"
ExecStart=/usr/local/promtail-vlogs/promtail-linux-amd64 -config.file=/usr/local/promtail-vlogs/promtail-config.yaml -config.expand-env=true
ExecStop=/usr/bin/kill -9 $MAINPID
Restart=on-failure
# CPU: hard limit of 1 core
CPUAccounting=true
CPUQuota=100%
# Memory: hard limit of 1 GB
MemoryAccounting=true
MemoryMax=1G
MemoryLimit=1G
MemorySwapMax=0
LimitNOFILE=65536
TasksMax=1024

[Install]
WantedBy=multi-user.target
EOF

Start Promtail

systemctl daemon-reload
systemctl enable --now promtail-vlogs
sleep 1
ss -lnt | grep '9081'
curl -s http://127.0.0.1:9081/ready
# Check whether data is being pushed
curl -s http://127.0.0.1:9081/metrics | egrep -i 'promtail_.*(client|sent|dropped|error|retry|targets)' | head -n 80

Grafana Configuration

This deployment option does not include Grafana. If Grafana is not available in your environment yet, please finish installing or upgrading Grafana first before continuing. The current reference VictoriaLogs datasource plugin version is 0.23.5, and Grafana must be >= 10.4.0. If your Grafana version does not meet this requirement, please first refer to Grafana 12.1.2 Upgrade Steps Reference.

Install the VictoriaLogs datasource plugin

cd /data/mdtemp
# Grafana >= 10.4.0; if not, upgrade Grafana first
wget https://pdpublic.mingdao.com/private-deployment/offline/common/victoriametrics-logs-datasource-v1.43.1_DATASOURCE.tar.gz
tar xzvf victoriametrics-logs-datasource-v1.43.1_DATASOURCE.tar.gz -C /usr/local/grafana/data/plugins/
systemctl restart grafana

Configure the datasource
- Set url to the VictoriaLogs address, for example http://127.0.0.1:9428.
- Turn on Basic auth.
- Set User to HTTP_AUTH_USERNAME.
- Set Password to HTTP_AUTH_PASSWORD.
- Keep Maximum lines at 2000000.
Import the slow query dashboard

This dashboard file is ready to use. Download it and import it into Grafana directly. If your environment already has its own dashboard version, you can also use it as a reference and make further adjustments.
Download or open the dashboard JSON
- Dashboard file: mongodb-VictoriaLogs-Slow-Query.en.json
- Import method: click Import dashboard in Grafana, then upload this file
- After importing, make sure the correct VictoriaLogs datasource is selected
- If you want to inspect the content first, you can also open the JSON file directly in your browser
Download the English dashboard JSON
Open the JSON file in a browser

Deploy VictoriaLogs​

Deploy Promtail​

Grafana Configuration​

Deploy VictoriaLogs

Deploy Promtail

Grafana Configuration