Deploy HDP

Prerequisites for Deployment

Create a new MySQL database: MDHDP

/usr/local/mysq/bin/mysql -h 127.0.0.1 -P 3306 -uroot -p123456

Create the database: MDHDP

CREATE DATABASE IF NOT EXISTS MDHDP DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci;

Deploy the Flink service, refer to: Flink Deployment Guide
- HDP functionality depends on the Flink service. Before enabling HDP features, ensure the Flink service has been successfully deployed.
- If Flink was previously deployed for Data Integration or Aggregation Table functionality, upgrade Flink to 1.17.1.530.
  
  Upgrade Notice
  After upgrading the Flink version, the existing data integration sync tasks and aggregation tables must be manually restarted to resume operation.
  - Flink cluster mode: Cross-Version Upgrade Guide
Download the HDP image

Download the HDP image on all nodes in the Kubernetes cluster where HAP microservices are running.
- Internet-accessible server
- Server without internet access
crictl pull registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hdp:1.0.0
Offline image download link. Download and upload to the deployment server after completion:
https://pdpublic.mingdao.com/private-deployment/offline/mingdaoyun-hdp-linux-amd64-1.0.0.tar.gz
Extract the offline image on the server:
gunzip -d mingdaoyun-hdp-linux-amd64-1.0.0.tar.gz
Import the extracted offline image:
ctr -n k8s.io image import mingdaoyun-hdp-linux-amd64-1.0.0.tar

Deploy HDP

In the directory where the microservices configuration files are stored, create a new file named hdp.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hdpwww
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hdpwww
  template:
    metadata:
      labels:
        app: hdpwww
      annotations:
        md-update: "20240123163208"
    spec:
      containers:
      - name: hdpwww
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hdp:1.0.0
        env:
        - name: ENV_SERVERID
          value: "single:hdpwww"
        resources:
          limits:
            cpu: "4"
            memory: 8096Mi
          requests:
            cpu: "0.01"
            memory: 64Mi
        readinessProbe:
          tcpSocket:
            port: 8380
          initialDelaySeconds: 10
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8380
          initialDelaySeconds: 180
          periodSeconds: 10

---

apiVersion: v1
kind: Service
metadata:
  name: hdpwww
  namespace: default
spec:
  selector:
    app: hdpwww
  ports:
    - name: http-hdpwww
      port: 8380
      targetPort: 8380

--- 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hdpapi
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hdpapi
  template:
    metadata:
      labels:
        app: hdpapi
      annotations:
        md-update: "20240123163208"
    spec:
      containers:
      - name: hdpapi
        image: registry.cn-hangzhou.aliyuncs.com/mdpublic/mingdaoyun-hdp:1.0.0
        env:
        - name: ENV_SERVERID
          value: "single:hdpapi"
        resources:
          limits:
            cpu: "4"
            memory: 8096Mi
          requests:
            cpu: "0.01"
            memory: 64Mi
        readinessProbe:
          tcpSocket:
            port: 8365
          initialDelaySeconds: 10
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8365
          initialDelaySeconds: 180
          periodSeconds: 10

---

apiVersion: v1
kind: Service
metadata:
  name: hdpapi
  namespace: default
spec:
  selector:
    app: hdpapi
  ports:
    - name: http-hdpapi
      port: 8365
      targetPort: 8365

Add the HDP startup command to the microservice startup script start.sh:

kubectl apply -f $baseDir/hdp.yaml

The modified start.sh script is as follows:

baseDir=$(dirname $0)

kubectl apply -f $baseDir/config.yaml
sleep 30
kubectl apply -f $baseDir/service.yaml
kubectl apply -f $baseDir/hdp.yaml  # Added this line
kubectl apply -f $baseDir/computinginstance.yaml
sleep 180
kubectl apply -f $baseDir/www.yaml

Add the HDP stop command to the microservice stop script stop.sh:

kubectl delete -f $baseDir/hdp.yaml

The modified stop.sh script is as follows:

baseDir=$(dirname $0)

kubectl delete -f $baseDir/www.yaml
kubectl delete -f $baseDir/service.yaml
kubectl delete -f $baseDir/hdp.yaml  # Added this line
kubectl delete -f $baseDir/computinginstance.yaml
kubectl delete -f $baseDir/config.yaml

Add the HDP restart command to the microservice restart script restart.sh:

sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/hdp.yaml

The modified restart.sh script is as follows:

RESTART_DATATIME=$(date +%Y%m%d%H%M%S)
baseDir=$(dirname $0)

sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/config.yaml
sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/service.yaml
sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/hdp.yaml  # Added this line
sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/datapipeline.yaml
sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/wps.yaml

# Commenting out the update of computinginstance.yaml
# sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/computinginstance.yaml

sed -r -i "/md-update:/ s/([[:digit:]]+)/$RESTART_DATATIME/" $baseDir/www.yaml

# Final message indicating a rolling restart
echo "Performing rolling restart for all services..."
/bin/bash $baseDir/start.sh

Add the following environment variable to config.yaml:
```
ENV_HDP_ENABLE: "true"
```
Restart the microservices for the changes to take effect.

Prerequisites for Deployment​

Deploy HDP​

Prerequisites for Deployment

Deploy HDP