Salesforce is your digital front tier core for your customer success and you integrate ERP system such SAP, webshops such as Amazon or eBay, payment systems such as PayPal, social media such as Facebook or LinkedIn, legacy data pool such as database, file or FTP to and from your Salesforce to get a connected and non-disrupted customer experiences using a middleware. It works as design but one day you realize that some data are missing which has been sent from the middleware. You are not sure which data was arriving on your Salesforce side and you ask yourself “Where can I see the data which was sent by the middleware?”. You are not asking the question as “Which data was sent from the middleware?” because the middleware provides you a nice monitoring feature to see data leaving the middleware platform.
You’ve connected an HR and billing system to your Salesforce and need to keep data history for your auditing department to prove the evidence for a government security and auditing check happened regularly at your company. For such an auditing process data on one side e.g. sender (middleware) is not sufficient and do not fulfill the government auditing standard and they request you to provide the other side e.g. data that arrive on the Salesforce side. How are you going to provide such logging and tracing of the historical data on the Salesforce side?
Since you use a middleware you have a nice feature to alert in case the message get failed while get sent to Salesforce. You love this feature because you don’t need to actively look every five minutes on the monitoring at your middleware. One day you realize that some of the alerts contained strange technical error text as
“java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:868)
at java.lang.StringBuilder.substring(StringBuilder.java:72)
at java.lang.AbstractStringBuilder.subSequence(AbstractStringBuilder.java:849)”
But you would expect a clear business error telling you something like “Quote xx cannot be posted due to invalid sales area code”. Which of the error text would you like to see?
Such situations can and will happen one day with your Salesforce integration and when it happens are you going to ask the same question instead of providing a solution to fix that? Read the following essay when you want to see the solution and answer.
How Middleware integration looks like
Let’s have a look and understand the Salesforce integration using middleware. To connect to Salesforce the middleware provides a technical connector which is able to invoke and consume the SOAP or REST API of Salesforce standard API. It uses the request/response pattern which is common to API principle and each call waits until the operation on the Salesforce side finish. This is the nature of synchronous API call which follows the tight coupling architecture pattern. With the standard Salesforce API, you can do great the basic four so-called CRUD operation meaning that you can create, read, update and delete a sObject.
Besides these four basic CRUD operation, there are no other additional capability and feature available on your Salesforce side. Some question you might ask is how you can do more than those four basic CRUD operation? Your integration, for example, needs to update a contact where it was changed on your ERP system such as SAP. But before the update, you need to check the assignment rule that the contact has the status ‘Released’ and is maintained by the sales agent group called ‘inside sales’. After you have updated the contact you want to do some cleanup operation to change the status to ‘Synchronized’. This means that you need more than only update the contact with some data which have been changed from SAP. You need pre- and post business logic to execute alongside the integration. How can you do such pre- and post additional business logic?
What is missing?
Because your Salesforce with all the apps from third-party vendors and your own app becomes such an important digital front tier layer for your custom success you want to have an enterprise-class integration which goes beyond the API connected approach. Refer to our other blog to understand the limitations and impact of the API only connected approach https://apsara-consulting.com/beyond-api/.
For enterprise-class integration, you need more than just the API connectivity. You might need to add business logic to your simple API integration. You might need monitoring, alerting and an easy way to handle errors on your Salesforce side. You might come to some urgent and emergency situation where the middleware is not alive and thus cannot resend the message to be processed on time. In this case, it would be a great help if you can modify data as you need on the Salesforce side and do a reprocessing. Furthermore, from the technical architectural standpoint, you need also a loose coupling pattern to handle mass data transfer which can only be done by using asynchronous API.
Why a decoupling services layer is needed
The additional value-added services mentioned above are needed for an enterprise-class integration which is not available on the Salesforce platform and which middleware cannot offer. To provide such services a decoupling layer is needed on the Salesforce platform which can stage the incoming data as messages for processing at any suitable time. Since no such a layer exists on the Salesforce platform SKYVVA has created such a layer to provide all the functionality like described in the above diagram.
The service layer is the technical basis for any functionality to allows the API processing to be executed in an asynchronous way. An asynchronous pattern is a crucial factor in the integration architecture because it provides the decoupling capability and independence of operation and thus is lose coupled. Lose coupled operation, on the other hand, has a big advantage and architectural benefits against the tight coupling pattern because it releases the client immediately from getting blocked. The client has not to wait until the invoked operation is finished. Less dependency and lose coupling create less cohesion and binding of interconnected operation which in term of resource consumption and performance is the best approach available for integration principle.
The loose coupling architectural style is just one of the outstanding feature resulted from having a service laver on the Salesforce side. Further benefits you have is, for example, the capability to shift the processing of the messages to a later time. Real-time and processing everything immediately sound great and fit best to some use cases where the user is waiting for a prompt response. But when coming to enterprise integration where the end-user is not involved you don’t need always a prompt response because it is useless and simply not needed.
Imagine that you have a webshop and you have to update on a daily basis millions of product prices. How can you do that at real-time? This is technically not possible doing in real-time because you will hit many Salesforce governor limits like the number of API calls per 24 hours, the available operating system thread to handle such a mass request and finally you will impact the performance of your Salesforce Org. and thus breaks the performance for your end-users in the business working hours.
In such a situation where it is sufficient to have the updated price of all product in the morning, you can use batch and bundling technique to send packages of for example 10k of price updates and process the messages in the night starting from 10:00 pm every day. Such scheduling is easy to configure using the scheduling functionality of SKYVVA. The benefit you get is shifting the processing of mass volume data into a time which is suitable for such a kind of data processing. Without a service layer like provided by SKYVVA, such a use case can simply not be handled by the Salesforce standard by just using the API connected approach only.
With a service layer on the Salesforce side, many additional value-added features like described above will be available to support you in your daily operation and maintenance of the integration scenario. The following description will show some important additional value-added feature in more detail.
Additional services you simply need
The various value-added feature will help and simplify the maintenance and operation of your daily integration with your Salesforce platform. Of course, you could say that you don’t need because everything works smoothly. But don’t forget the one day where some issues will arise and troubles start to appear sooner than you ever can think. I this situation you will not want to miss the additional value as the following examples below.
- Monitoring
Give you a second eye to see what happened on the Salesforce side. With the additional monitoring layer, you have a real end-2-end monitoring and have 100% view on both sides of the connected platform. - Reprocessing
Give you a possibility to independently do reprocessing of failed messages in case the middleware is not alive and can’t resend the data. This is to fix an emergency case where you have to make a quick fix inside Salesforce to reprocess the data which get failed due to some business data inconsistency. - Alerting
Give you real business error alert which deals with error happened on your Salesforce side and not an error which happened on the middleware. The alert is sent from the Salesforce side and therefore is a real and authentic alert cover the business data error on your Salesforce platform. - Workflow
Let you define a rule and pre-condition to process the message so that you can block wrong and dirty data from being mess up your business data on the Salesforce side. Complex business validation can run as a pre-condition before the data get posted and after the data get posted you can do a post-processing logic using the SKYVVA workflow. - Business logic
When you need to add business logic to the data posting Salesforce give you great and flexible tool like apex, trigger, process builder and flow. Use one of them to add business logic to your pure data processing linked to the SKYVVA interface. - Batch and Scheduling
When your data synchronization requirement comes to meet a mass volume data updates it is not correct and technically possible to use a real-time approach. In this case, you should move the processing of these mass data to a later free time window where you don’t break down the performance of the Salesforce Org. and thus impact the end-user while their business working hours. Instead, you should shift the processing to a suitable time for example in the night starting from 10:00 pm.
There are more additional value-added feature SKYVVA have added as integration services to the Salesforce platform. To keep this blog clear, short and comprehensible only these six value-added features will be explained in more detail.
Value-added #1 - Monitoring
To showcase the monitoring value-added feature we need to use a real interface example where the connected systems are SAP-ERP (sender), SAP-PI (middleware) and your Salesforce (receiver). The scenario is to change customer data in SAP and the changes will be synchronized immediately back to Salesforce. The diagram below shows the example flow.
In this screen, we can see the data changed in SAP for the customer 301557 and the old address in Salesforce. We expect to have the changed address from Salesforce to be updated on the Salesforce side.
Now have a look at the monitoring within the connected system as shown in the below picture. In this example, we are integrating to Salesforce using the standard API approach. In this screen, we can see the different monitoring on the sender e.g. SAP-ERP, on the middleware e.g. SAP-PI but on the Salesforce side there is no monitoring capability with the Standard Salesforce API. What you can do is going to check directly the sObject Account to see if the address data has been changed. When you check the address on the account you don’t see the changes getting updated in Salesforce. So what happened and what can you do?
What you can do in this case is to go to the debug trace and try to find the root cause why there the address has not been updated. Probably you will find something, probably not. As the Salesforce user or administrator, you cannot complain to the SAP-PI middleware team or the SAP-ERP team because all of them will prove you through their monitoring that the data was sent successfully where you can see it with the green flag. So what can you do to find the root cause in an easy way?
Unfortunately using the standard API approach with the middleware there is no monitoring capability inside Salesforce. The only way is to get your skilled apex developer team to try to find the root cause for this issue. Note that in this case we are using the asynchronous technique on SAP-ERP with Idoc and on SAP-PI the interface was designed as an asynchronous interface as well. There is no need to design this scenario using a synchronous interface because the SAP Idoc is an asynchronous technique. Therefore there will be no response of the API call telling the sender e.g. SAP-PI any error or exception. The message was sent successfully from SAP-PI to Salesforce and due to the nature of the asynchronous processing no further steps are needed on SAP-PI side.
So how to overcome this situation? Now let’s have a look at the same scenario which we rebuilt with SKYVVA. You are using, in this case, the SKYVVA connector on the SAP-PI/PO side. On the Salesforce side, you are using the SKYVVA service layer.
When the message arrives from SAP-PI/PO middleware it goes to the SKYVVA message layer where it gets processed immediately. From the message flow, there is no different to the scenario where SKYVVA was not involved. The significant differences are that now you have the SKYVVA layer which can provide you the monitoring service. So how does it looks like?
As you can see in the screen we have also monitoring in Salesforce where you can immediately see the root cause what was happened. The error is clear and understandable for the Salesforce user and administrator. Now you find the error in minutes instead of hours without involving your apex developer team. This is a real End-2-End monitoring enhancing your middleware integration scenario.
Value-added #2 - Reprocessing
Let’s stay with the above example which we used to showcase the monitoring. Let have a look at the red message in the SKYVVA monitoring. Let’s assume that you as the Salesforce user or administrator understand the business error clearly and you need to get the address change because you have to send the quote to that customer today before the end of the business day. You cannot wait until the next day to risk to lose the deal. You take your phone and ring your SAP-ERP colleague to resend the incorrect message again. You reach him but unfortunately, he left the office already because it was short to the office closing hour. Your SAP-ERP colleague cannot resend it today and promise to send tomorrow earlier in the morning. But this will not help you because tomorrow morning is too late for your emergency situation now. So what can you do?
Since you have rebuilt the interface with SKYVVA just use the reprocessing feature. Open the message, change and save the data and do a reprocessing.
Click on the button ‘Reprocess’ and confirm by clicking on the button ‘Yes’.
Now you can see a green message after the reprocessing.
Now you can navigate from here directly to the business record to see the change made for the address update.
The new address gets updated now after the reprocessing and you can use the new address to send the quote through your automatic quote process. Note that due to the simplicity the and clarity for the demonstration of the reprocessing feature we are not going to send the changes back to SAP. This is possible and makes sense but it would stretch the writing too long just for showing the reprocessing feature only. Also, note that the reprocessing you see here is the manual reprocessing. For this example, you have to do the manually reprocessing because the data is wrong and because you need to correct the data. But there is also others error where you don’t need to do a manual reprocessing but let the reprocessing job do it which you can schedule in an easy way. Have a look at the scheduling feature of the reprocessing job.
This is the message reprocessing job which you easily can start and stop by demand.
Value-added #3 - Alerting
The monitoring feature is great to see what happened on both sides. The annoying thing of the monitoring is that you always have to actively look to it. Wouldn’t it be great if you have some alert or notification when something gets failed and you get them as mail?
When looking to the scenario above SAP-ERP has an alerting, SAP PI PO has an alerting but on Salesforce there is no alert provided by the standard API. You would say that it is enough to have the alert on the middleware. Let’s have a detail look at the alert situation.
An alert should give you the real reason and root cause of an error which can happen in different places at any time. The error can be happened on SAP-ERP, on SAP PI PO, and on Salesforce and you need to have the original error text. In the above situation, the middleware alert the error happened on the middleware and not really what happened on Salesforce. It can reflect the error from Salesforce but there is a risk with this approach. The middleware has its own logic and code which can malfunction while transferring the message to Salesforce. In this case, it would alert its own error inside the middleware as for example “java.lang.StringIndexOutOfBoundsException: String index out of range: -1” which has nothing to do with the real error happened on Salesforce side.
What is the problem here? The alert is not authentic and it could be simply wrong because you get an alert which tells you the issue or error happened on the middleware but not happened on the Salesforce side. If the middleware stuck for some reason you could get alert which is delayed and was happened in Salesforce a few hours ago. This is crucial when you have a real-time scenario which also needs real-time alert from the source of truth. With the middleware in between, it is a kind of alert forwarding from Salesforce to you but not an alert delivering from Salesforce directly to you. From this example, you can see some drawback of the alerting capability.
- Inaccurate alert and sometimes simply the wrong alert
- Not real-time enough in case the middleware stuck or its performance gets degraded
- Not direct but indirect through forwarding by the middleware
How can we overcome these issues? The only accurate approach to deliver alert on time, correct and with its original root cause is to have the alerting component directly in Salesforce as well. As you can see in the picture above each component e.g. SAP-ERP and SAP PI PO has its own alert component. There is a good reason to have a native alert component inside each platform. Now let’s see how we can have such a native alert component inside Salesforce.
SKYVVA has added with its service layer the native alert component to be able to deliver the true alert. Alerts are directly sent real-time from Salesforce to you without any delay. You will get the original error text and never see something like:
“java.lang.StringIndexOutOfBoundsException: String index out of range: -1”.
Alerting can be defined in a granular way based on the integration or interface level. The integration is a logical group of interfaces. There are four different way to define alert delivery as below:
- Using external mail
- Using internal mail link to a user
- Using task
- Using chatter group
Here is an example, email alert.
You can click on the link and jump directly to the message detail and reprocess that failed message. With the option to use the chatter group you can use the social media feature to organize your alerting delivery. People following the group will get alert and there is no complex setup procedure to follow. There are also other alerting features for specific SKYVVA functionality like the Batch and CDC processing to observed the inbox of the basket or the change pointer table for data change.
Value-added #4 - Workflow
When you are using the Salesforce standard API to CRUD e.g. Create, Read, Update or Delete a sObject there is no validation or checks before create, update or delete a business record. It just above picture) which a service layer could bring to your integration need. It goes beyond the connectivity and provides you and your Salesforce team an easy way to develop, maintain and operate your daily integration need. This is what the SKYVVA service layer adds to the Salesforce Lightning platform which is missing for having the right and robust tool for doing the integration to and from Salesforce.executes it and modifies the sObject record immediately. But in most integration scenario you need to have some validation before modifying your business record otherwise you could mess up and create data inconsistency.
Have a look to the example which can be solved easily using the built-in SKYVVA workflow to have as a pre-condition check before an Asset gets modified. Here different condition consisting of the complex expression with formulas can be defined. A chain of ruled-based interface processing can be set up and provide you a powerful way to add complex message processing behavior than the simple CRUD operation. You can define any complex rule to validate first before you modify the sObject.
With the workflow, you cannot only define a rule or condition for interface processing. You can define a flow containing multiple steps involving in the message processing. Let’s assume you want to modify the account and the contact after each other. In this case, you can define a workflow which receives the account and contact data in a message and passing this message to the account and contact interface processing step. Thus you have defined a chain of processing step which can handle complex operation in one single API call.
You can define the different setting and run-time behavior for the workflow engine. For example, you can design a transaction-based processing flow and rollback all the operation within the flow when an error occurred. This guarantee transactional consistency.
Value-added #5 – Business logic
A message is transferred as the payload from the middleware to your Salesforce and through the Salesforce standard API, you can CRUD (Create, Read, Update, Delete) a sObject. The CRUD operation is the basic and elementary action you can think of handling with the business record in Salesforce. Sometimes it is enough just to CRUD sObject when your requirement is simple. Sometimes it is not enough when your integration scenario should handle some business requirement which goes over the CRUD operation.
With the SKYVVA workflow to define rules-based interface processing. Here you can use the built-in capability from Salesforce to add the business logic to the message processing. When your integration scenario goes beyond the CRUD operation then you need a powerful technique to add your business logic as you can do with a programing language. In this case, you can link an apex class to the interface and when the message arrives at the SKYVVA layer the processing will execute with your apex class logic where you can implement your business logic.
You can utilize the available programming and declarative tool for adding the business logic to your data integration. If you are a developer and can code in apex you can use apex trigger and apex class. If you are an admin and not an experienced developer you can use non-declarative tools such as the Process Builder and Flow. From the interface, you can link a Salesforce flow of type ‘Autolaunched’ and can build the business logic without any coding using the graphical flow.
Value-added #6 – Batch and Scheduling processing
Integration using middleware in between is mostly used to connect application to application scenario. This means that real-time scenario with end-user sitting and waiting on the other side is not common and thus not always a real-time and immediate data processing are needed. What sometimes need for mass and high-volume data integration between application system are batch and background processing to optimize the resource and performance and utilize the network, cpi and data storage.
Based on the integration scenario some use case makes perfect sense to use the real-time processing pattern. For example (1st scenario) when a user on the Salesforce side want to see the availability of a product in SAP-ERP then it makes sense to invoke an API and wait for the response. Such request is short running operation and will not hit the Salesforce time-out limit. When you want to update price by a nightly operation (2nd scenario) then it makes no sense to use a real-time processing pattern to update millions of prices. This would be technically not possible because of the Salesforce timeout limit and secondly make no sense because no user sits and wait to see the updated price immediately. It is enough to have the price update in the next day when the webshop will be open.
In fact, it is not a question whether the real-time (tight coupling) or batch/background driven (loose coupling) approach is better than the other. Both patterns fit perfectly when the given scenario is suitable. Do it real-time or batch based on your business requirement. The problem with the standard API is that it is not possible to delay the processing of the API or push it to run in the background job at a later time as for example in the night at 10:00 pm every day. The mechanism to do that is simply missing with the Salesforce standard.
SKYVVA service layer adds a scheduler and job processing component to the Salesforce platform which provides the batch and scheduling processing.
You can define the schedule in a different way to meet your business requirement. You can run every for example 15 minutes. You can run once at a certain time e.g. at 1:00 am or you can run every hour between for example 10:00 pm and 4:00 am.
You can select the days in the week and the month you want to exclude. This is a flexible way to define when message processing should start based on your requirement. You are not forced to process everything immediately if it would make more sense to process the data like a mass price update at the night time. Thus you can adjust and distribute the processing load and utilize your Salesforce resources and CPU time in a better way.
Different schedules are available for any kind of task you needed. For example with the batch processing and reprocessing scheduler you can push packages of message e.g. 10k per package and schedule the processing into the night time from 10:00 pm to 6:00 am for every hour. This utilizes the resources and distribute the load and keep the performance for of your Salesforce instance for the end-user while their working hours smoothly.
Starting and stoping schedule is easy by just clicking on the icon . Setting the schedule only need to click on the icon . All available schedule for the different task is generated automatically.
Summary
All six described value-added features will raise your integration quality to the next higher level giving you the operational instruments to ease and simplify the daily task. This method goes beyond the simple API integration approach where only pure connectivity is possible. When you want an enterprise-class integration then the only middleware approach is definitely not enough.
End-2-End monitoring and alerting increase the quality and improve error handling and daily operation tremendously. Message reprocessing give you a handy tool to handle emergency cases where the sender system is currently out of order. With workflow and business logic the simplicity and limitation of the CRUD operation can be overcome and complex business scenario can be easily implemented without much coding on the wrong side. Lastly, batch and scheduling feature enables to decouple the sender from the receiver and thus create less dependency which leads to optimized performance in the message processing for mass volume data.