Deployment Architecture

In today’s dynamic enterprise landscape, organizations face increasing complexity when integrating on-premises systems with cloud-based services. Key challenges include:

• Unstable connectivity between geographically distributed on-premises environments and cloud infrastructure
• Security vulnerabilities in data transmission and access control
• Limited scalability to handle fluctuating workloads efficiently
• Lack of centralized monitoring, making proactive issue resolution difficult
• Risks of data integrity and availability due to single points of failure

To address these challenges, LDAS AWS Architecture offers two deployment models tailored to your business needs:

AWS - High Availability

The diagram depicts a high availability architecture designed to handle varying loads efficiently.

1. On-premises agents and applications across multiple customer sites connect securely to LDAS Cloud services on AWS using network firewalls or AWS Direct Connect, ensuring encrypted data transmission and protection against unauthorized access.
2. The cloud infrastructure includes Amazon EKS for container orchestration, EC2 Auto Scaling Groups behind a Network Load Balancer for dynamic traffic distribution, and Amazon EFS for scalable shared storage across availability zones.
3. Data services are highly available with Amazon RDS for transactional and configuration data, Amazon DocumentDB for semi-structured data, and Amazon S3 for storing static assets, logs, and raw instrument data.
4. Security is enforced through IAM roles, policies, and security groups to tightly control access and maintain data confidentiality across the hybrid environment.
5. Monitoring and observability are implemented within the cluster using Prometheus to scrape metrics from Loki, while Grafana provides real-time visualization and alerting for system health and performance.

AWS - Standard Deployment

This section includes users and applications that interact locally within the on-premise environment. It represents the initial point of interaction where data is generated and processed before being securely transferred to the cloud.

1. A secure connection between the on-premises environment and AWS Cloud is established via a network firewall or AWS Direct Connect, ensuring encrypted data transfer and protection from unauthorized access. The application server uses a microservices architecture for scalability and flexibility, enabling efficient handling of low to medium workloads with consistent performance.
2. EC2 Instances host application components and agent services, while Docker is used to containerize and run LDAS services. Amazon Elastic File System (EFS) provides shared storage, Amazon S3 handles object storage and archival, and Amazon Elastic Container Registry (ECR) stores Docker images securely.
3. The database layer includes Amazon RDS for structured relational data with automated backups, and Amazon DocumentDB for document-based data—ensuring reliable, scalable, and accessible data management.

On-Premises Standard Deployment

This section outlines the standard on-premises deployment setup, where instruments and applications operate within a secure local environment. It highlights how data is collected, processed, and securely transmitted through containerized services and persistent storage components, ensuring reliable performance and system integrity.

1. Instruments and applications transfer data securely to the LDAS application through a Secured Gateway, ensuring authenticated connectivity and protection against unauthorized access. The LDAS Agent Server interfaces with these instruments and applications to facilitate data ingestion.
2. The App Server is built using containerized microservices and hosts LDAS Microservices via Docker Compose. It includes:
   • Redis for caching
   • Prometheus and Grafana for monitoring
   • Camunda for workflow orchestration
   • NiFi for managing data flow
3. The DB Server (Statefulset Server) manages persistent data storage and streaming using a robust technology stack:
   • PostgreSQL and MongoDB support relational and NoSQL data models respectively.
   • Kafka enables distributed event streaming for real-time data flow
   • Elasticsearch powers search and analytics across large datasets

Communication between the App Server and the database layer is encrypted using TLS (Transport Layer Security), ensuring data integrity, confidentiality, and secure interactions across the system.

On-Premises Native Cluster Deployment

This section describes the native cluster-based on-premises deployment, designed for secure, scalable, and efficient data orchestration. It highlights how instruments, agent servers, and clustered components work together to manage data flow, processing, and storage within a controlled local infrastructure.

1. A secure connection is established between instruments and the LDAS Agent Server via port 443. This ensures authenticated communication and protects data transfer within the on-premises environment.
2. The system is deployed as a native cluster with a master node and multiple worker nodes.
   • The master node orchestrates tasks, manages workloads, configurations, and monitors cluster health.
   • Worker nodes are horizontally scalable based on system load and host essential services including LDAS Microservices, PostgreSQL, MongoDB, Camunda, Apache NiFi, Redis, Kafka, and Elastic.
   • Monitoring and observability are enabled using Prometheus and Grafana.
3. To maintain optimal performance and responsiveness, the number of worker nodes is configured based on expected system load. This flexible scaling approach ensures efficient resource utilization and high availability within the on-premises infrastructure.
4. Built-in redundancy, failover mechanisms, and backup strategies ensure resilience and uninterrupted service.
5. The database layer includes PostgreSQL and MongoDB, supporting both structured and unstructured data storage respectively.
6. Communication between the application and database layers is encrypted using TLS (Transport Layer Security), ensuring data integrity, confidentiality, and secure interactions.

LDAS Hosting Options for Enterprise Environments

LDAS is designed with enterprise-grade flexibility, offering multiple deployment models to align with diverse IT strategies, compliance needs, and operational preferences.