Deploy HDP

Prerequisites for Deployment

1. Create a MySQL database: MDHDP

   1. Log in to MySQL:

      /usr/local/mysq/bin/mysql -h 127.0.0.1 -P 3306 -uroot -p123456

   2. Create the database: MDHDP

      CREATE DATABASE IF NOT EXISTS MDHDP DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci;

2. If MongoDB authentication is enabled, you need to manually create the mdhdp database. For details, refer to How to Create a MongoDB Database

3. Please enable the aggregation table feature in advance. Refer to How to Enable the Aggregation Table Feature

4. Deploy the Flink service, refer to: Flink Deployment Guide

   • HDP functionality depends on the Flink service. Before enabling HDP features, ensure the Flink service has been successfully deployed.

   • If Flink was previously deployed for Data Integration or Aggregation Table functionality, upgrade Flink to 1.19.720.

     Upgrade Notice

     After upgrading the Flink version, the existing data integration sync tasks and aggregation tables must be manually restarted to resume operation.

     • Flink cluster mode: Cross-Version Upgrade Guide

5. Download the HDP image

   Download the HDP image on all nodes in the Kubernetes cluster where HAP microservices are running.

   Internet-accessible server

   crictl pull registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hdp:1.1.0

   Server without internet access

   Offline image download link. Download and upload to the deployment server after completion:

   https://pdpublic.mingdao.com/private-deployment/offline/mingdaoyun-hdp-linux-amd64-1.1.0.tar.gz

   Extract the offline image on the server:

   gunzip -d mingdaoyun-hdp-linux-amd64-1.1.0.tar.gz

   Import the extracted offline image:

   ctr -n k8s.io image import mingdaoyun-hdp-linux-amd64-1.1.0.tar

Deploy HDP

1. In the directory where the microservices configuration files are stored, create a new file named hdp.yaml with the following content:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hdpwww
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: hdpwww
     template:
       metadata:
         labels:
           app: hdpwww
         annotations:
           md-update: "20240123163208"
       spec:
         containers:
         - name: hdpwww
           image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hdp:1.1.0
           env:
           - name: ENV_SERVERID
             value: "single:hdpwww"
           resources:
             limits:
               cpu: "4"
               memory: 8096Mi
             requests:
               cpu: "0.01"
               memory: 64Mi
           readinessProbe:
             tcpSocket:
               port: 8380
             initialDelaySeconds: 10
             periodSeconds: 10
           livenessProbe:
             tcpSocket:
               port: 8380
             initialDelaySeconds: 180
             periodSeconds: 10
   
   ---
   
   apiVersion: v1
   kind: Service
   metadata:
     name: hdpwww
     namespace: default
   spec:
     selector:
       app: hdpwww
     ports:
       - name: http-hdpwww
         port: 8380
         targetPort: 8380
   
   --- 
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hdpapi
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: hdpapi
     template:
       metadata:
         labels:
           app: hdpapi
         annotations:
           md-update: "20240123163208"
       spec:
         containers:
         - name: hdpapi
           image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hdp:1.1.0
           env:
           - name: ENV_SERVERID
             value: "single:hdpapi"
           resources:
             limits:
               cpu: "4"
               memory: 8096Mi
             requests:
               cpu: "0.01"
               memory: 64Mi
           readinessProbe:
             tcpSocket:
               port: 8365
             initialDelaySeconds: 10
             periodSeconds: 10
           livenessProbe:
             tcpSocket:
               port: 8365
             initialDelaySeconds: 180
             periodSeconds: 10
   
   ---
   
   apiVersion: v1
   kind: Service
   metadata:
     name: hdpapi
     namespace: default
   spec:
     selector:
       app: hdpapi
     ports:
       - name: http-hdpapi
         port: 8365
         targetPort: 8365

2. Add the HDP startup command to the microservice startup script start.sh:

   kubectl apply -f $baseDir/hdp.yaml

   The modified start.sh script is as follows:

   baseDir=$(dirname $0)
   
   kubectl apply -f $baseDir/config.yaml
   sleep 30
   kubectl apply -f $baseDir/service.yaml
   kubectl apply -f $baseDir/hdp.yaml  # Added this line
   kubectl apply -f $baseDir/computinginstance.yaml
   sleep 180
   kubectl apply -f $baseDir/www.yaml

3. Add the HDP stop command to the microservice stop script stop.sh:

   kubectl delete -f $baseDir/hdp.yaml

   The modified stop.sh script is as follows:

   baseDir=$(dirname $0)
   
   kubectl delete -f $baseDir/www.yaml
   kubectl delete -f $baseDir/service.yaml
   kubectl delete -f $baseDir/hdp.yaml  # Added this line
   kubectl delete -f $baseDir/computinginstance.yaml
   kubectl delete -f $baseDir/config.yaml

4. Add the HDP restart command to the microservice restart script restart.sh:

   sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/hdp.yaml

   The modified restart.sh script is as follows:

   RESTART_DATATIME=$(date +%Y%m%d%H%M%S)
   baseDir=$(dirname $0)
   
   sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/config.yaml
   sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/service.yaml
   sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/hdp.yaml  # Added this line
   sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/datapipeline.yaml
   sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/wps.yaml
   
   # Commenting out the update of computinginstance.yaml
   # sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/computinginstance.yaml
   
   sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/www.yaml
   
   # Final message indicating a rolling restart
   echo "Performing rolling restart for all services..."
   /bin/bash $baseDir/start.sh

5. Add the following environment variable to config.yaml:

   ENV_HDP_ENABLE: "true"

   If MySQL is in MGR mode, add the following configuration

   If MySQL is deployed in MGR mode, add the following environment variables to the HAP microservice config.yaml file to connect to the MySQL Router:

   ENV_HDP_MYSQL_HOST: "192.168.10.21"
   ENV_HDP_MYSQL_PORT: "6446"
   ENV_HDP_MYSQL_USERNAME: "root"
   ENV_HDP_MYSQL_PASSWORD: "123456"

   ENV_HDP_MYSQL_HOST only needs to be set to any Router IP address. ENV_HDP_MYSQL_PORT defaults to 6446 (the Router read/write port), and the username and password should be replaced with the actual connection credentials.

6. Restart the microservices for the changes to take effect.