How to manage data from same microservice on different AWS infrastructure

Currently we have a micro-service (Java based) deployed to fetch, modify and save data on AWS infrastructure, which consists of a data lake.

Our AWS based data lake consists of S3, Kinesis, Athena, RDS, Airbyte connectors, Airflow, etc. We have designed our AWS data lake to store, process and manage data of 100 different clients, which are identified by app_id.

If the number of applications increase more than 100, we create a new AWS infrastructure with same resources for data lake and Java based micro-service.

We have created a client UI which interacts with micro-service to see final statistics of data stored in data lake after it is processed.

Now the problem is that UI is deployed on a separate infrastructure. If admin users wants to view data from 1 to 100 app_ids, it should call micro-service deployed on host1. If the data is needed from 101 to 200, host2 micro-service should be called.

Please advise on how to achieve this. Based on app_id, the server should redirect the REST call to correct host.

One way that I think of is that each host should maintain the in-memory map of all the hosts and their app_ids. This in-memory map will be maintained on each host. We will create a separate endpoint for UI which it can query and pass app_id. Based on app_id, the micro-service (backend) will return a hostname and then UI will call the actual API
deployed on that host name. The code for the same is as follows:

@Override
    public HostApplicationMappingsResponseDTO getHostForApplicationByIdAndName(String applicationId, String applicationName,
           boolean isRefreshRequired, String token, String origin) throws IOException {
        LOGGER.info(" Entering in getHostForApplicationByIdAndName {}", applicationId);
        ApplicationDTO applicationDTO = clusterCloudRepository.getHostForApplicationByIdAndName(applicationId, isRefreshRequired, token, origin);
        if (applicationDTO == null) {
            return new HostApplicationMappingsResponseDTO(new com.firm.controller.dto.Error(HttpStatus.NOT_FOUND.value(),
                    HttpStatus.NOT_FOUND.getReasonPhrase(), "Application not found", null));
        }
        return applicationDTO;
    }

    @Override
    public FirmApplicationListResponse getApplicationByHost(String token, String origin) throws IOException {
        LOGGER.info(" Entering in getApplicationByHost {}");
        FirmApplicationListResponse response = new FirmApplicationListResponse();
        List<ApplicationDTO> appList = clusterCloudRepository.getApplicationsByHost("", token, origin);
        if (CollectionUtils.isEmpty(appList)) {
            return new FirmApplicationListResponse(new com.firm.controller.dto.Error(HttpStatus.NOT_FOUND.value(),
                    HttpStatus.NOT_FOUND.getReasonPhrase(), "Applications not found", null));
        }
        response.setApplications(appList);
        return response;
    }

This approach results in 2 REST calls. Is there any way to achieve in a single call?

Please advise on this.

Leave a Comment